Download S3 files using PowerShell

Download or copy S3 files

Use the below script to download the files from any S3 bucket to your local machine

$sourceBucket = '<bucket name from where you want to copy the files from>'
$profile = '<aws profile name>'
$Folder = '<Folder Path on local drive>'

$items = Get-S3Object -BucketName $sourceBucket -ProfileName $profile -Region 'us-east-1'
Write-Host "$($items.Length) objects to copy"
$index = 1
$items | % {
    Write-Host "$index/$($items.Length): $($_.Key)"
    $fileName = $Folder + ".\$($_.Key.Replace('/','\'))"
    Write-Host "$fileName"
    Read-S3Object -BucketName $sourceBucket -Key $_.Key -File $fileName -ProfileName $profile -Region 'us-east-1' > $null
    $index += 1
}


Disclosure: We use affiliate links to monetize our content.  We may receive a commission on products or services that you purchase through clicking on links within this blog.

Building APIs with API Gateway

API Gateway is one of the managed services provided by AWS to create, publish, monitor and secure REST and WebSocket APIs.

Serverless development allows the developer to focus more on the application code and logic and not worry about the infrastructure details.

Scaling which is handled by AWS also becomes simpler as the developer only has to invoke the APIs in response to events. There are no more idle servers sitting and waiting for the events to trigger as AWS manages the availability of servers as and when required.

The key benefits of serverless development is –

  • No infrastructure management
  • No idle servers
  • High availability
  • Flexible scaling.

There are a number of managed services provided by AWS that act as the building blocks for your application. Some of them are AWS Lambda that provide you with compute services, S3 for storage services, Amazon Aurora for the relations database needs or Dynamo DB for NoSQL databases.

All the interaction with these managed services provided by AWS involve invoking an API and getting the response back which mainly consist of the business logic you need got your application.

The service which we are going to discuss further is API Gateway which is designed to help you develop and operate the APIs.

 

API Gateway

API Gateway is one of the managed services provided by AWS to create, publish, monitor and secure REST and WebSocket APIs. The REST APIs created using API Gateway are HTTP-based and stick to the REST protocol which allows for stateless client-server communication.

You can implement the standard HTTP methods like GET, PUT, POST, DELETE, and PATCH.

You can simply access the API Gateway from the AWS Management Console and if you are using one of the available programming languages in AWS SDKs, then SDK could be used to access the API Gateway. It can also be accessed using the AWS Command Line Interface as well.

 

How API Gateway works

api gateway architecture In the above diagram, on the left we have all users, web and mobile applications, IoT devices and all the private applications that are on VPC or on-premises. All these connect to the API Gateway using the HTTP-based request

API Gateway receives the request and determine what to do with that request. API Gateway adds a CloudWatch monitoring where you can easily monitor the API calls. It also determines if the response from the application is cached in the memory (API Gateway Cache).

API Gateway then invoke a number of backend managed services like AWS Lambda, EC2 endpoints, etc.

 

Choosing the right API endpoint (Host name of the API)

The selection of the right API endpoint depends on where the majority of your application traffic originates. There are basically 3 types of API endpoints –

  • Edge optimized – It is helpful to reduce client latency as the API requests are routed to the nearest CloudFront Point of Presence (POP).
  • Regional – It is helpful to reduce latency when the calls are triggered from the same region as the API
  • Private – It is helpful when you want to only expose the APIs inside your VPC.

Edge Optimized

edge optimized API Gateway deploy a CloudFront distribution which is managed by API Gateway. The nearest CloudFront receives the request from the customer and route it to API Gateway endpoint in the AWS region. This is how the customer latency is reduced.

 

Regional

 

 

This is completely different than Edge Optimized end point that we just discussed above. In this there is no CloudFront distribution deployed by AWS. The customer request directly goes to the API Gateway that are in a specific AWS region. This endpoint provides lot of flexibility as you can customize different scenarios like load balancers, etc.

You can decide how the request from the customer is routed towards the endpoint. You can also add firewalls, caching and SSL certificates as well with this style.

 

Private

 

 

This endpoint is only exposed from within the VPC. It means that all the API requests originates from within your own VPC and they are not exposed publicly.

This type of API endpoint is used for secure work environments like financial institutions, etc.

 

It is very important to select the right API endpoint from the above 3 types. You can easily choose the type of endpoint you want to use from the dropdown while creating the API. If your requirements change later on, you can simply come back and change the API and you are not locked down to only one type of endpoint.

AWS Firecracker

AWS announced Firecracker in 2018, which is a new virtualization and open source technology developed using the language Rust.

AWS Lambda was launched few years back and it provided a secure serverless experience to the developers where they don’t have to manage the severs and completely avoid managing the infrastructure as well.  Lambda currently processes trillions of executions at a time for over thousands of customers. Learn more about AWS Lambda here and get your hands dirty!

In 2018, AWS launched Fargate that extended the benefits of serverless to containers and is currently running millions of containers for the AWS customers.

Firecracker Introduction

AWS announced Firecracker in 2018, which is a new virtualization and open source technology developed using the language Rust.

It is basically a new way to enhance the backend implementation of both Lambda and Fargate. This new technology makes the use of KVM (Kernel-based Virtual Machine) and provide RESTful API to launch secure and lightweight micro virtual machines in non-virtualized environments. It combines the speed, efficiency and performance by containers with the security offered by the virtual machines.

Firecracker runs on Intel processors today, with support for AMD and ARM coming in 2019. It is open sourced under Apache 2.0 and supports Linux host and guest operating systems with kernel versions 4.14

Key features –

  • Secure – It uses multiple layers of isolation that provides enhanced security over traditional virtual machines.
  • Less overhead – You can run thousands of secure VMs with any combination of vCPU and memory to match the application requirement.
  • Speed – It accelerates loading of kernel with a minimal configuration that enables fast startup times. You can launch a microVM in less than 125ms.
  • Open source – It is an open source project. You can find the git repo here
  • Efficiency – Each microVM runs with a reduced memory overhead of 5MiB.



Source – https://aws.amazon.com/blogs

Fetch json data in React

How to fetch JSON data in React?

Sample json data (sampledata.json) – 

[
 {“id”: “1”,”name”: “Sam”},
 {“id”: “2”,”name”: “John”},
 {“id”: “3”,”name”: “Alex”},
 {“id”: “4”,”name”: “Jack”}
]
React component to read the above json data – 

import React, { Component } from 'react';
var myRequest = new Request('sampledata.json');

export default class App extends Component {
render(){

fetch(myRequest)
   .then(function(response) {
     if (!response.ok) {
        thrownewError("HTTP error, status = "+response.status);
     }
     returnresponse.json();
    })
   .then(function(json) {
     for(vari=0; i&amp;lt;json.length; i++) {
       console.log(json[i]);
     }
   });
   return (
<div></div>
<pre>)
   }
}

What to expect in C# 8.0

C# 8.0 is the next major release version of C#

C# 8.0 is the next major release version of C#. As the work is still in progress and the below list of features is not a complete list, but it will definitely give you an idea of what you will be getting from the next major release.

New Features in C# 8.0

  • Nullable Reference Types

This is one of the cool features that you will love to have in the new version of C#.

This feature will allow the developers to:

  1. Express whether a variable, parameter or result of a reference type is intended to be null or not.
  2. Provide warnings when such variables, parameters and results are not used according to that intent.

Till now, you cannot add null into ordinary reference types such as string. Now with this new feature, the developers will only get warnings, not errors.


string s = null; // Warning: Assignment of null to non-nullable reference type

But, if you really want to have null, then use a nullable reference type, such as string?


string? s = null;

A flow analysis tracks nullable reference variables and the compiler will analyze the flow of your code to see if a null value could make it to where you are using it.


void M(string? s)
{
   Console.WriteLine(s.Length); // Warning: Possible null reference exception

   if (s != null)
   {
       Console.WriteLine(s.Length); // Ok: You won't get here if s is null
   }
}

  • Recursive Patterns Matching

Pattern matching extensions for C# enable many of the benefits of algebraic data types and pattern matching from functional languages, but in a way that smoothly integrates with the feel of the underlying language.


IEnumerable GetEmployeesOnLeave()
{
    foreach (var e in Employees)
    {
      if (e is Employee { OnLeave: true, Name: string name }) yield return name;
    }
}

The pattern Employee { OnLeave: true, Name: string name }) checks that the Person is a Employee, then applies the constant pattern true to their OnLeave property to see if they’re on leave, and the pattern string name to their Name property to get their name (if non-null). Thus, if e is a Employee, is not on leave and has a non-null name, you will get the name of those employees.

  • Async Streams

Until now, C# has support for iterator methods and async methods, but no support for a method that is both an iterator and an async method. C# 8.0 is coming with this new feature by allowing for await to be used in a new form of async iterator, one that returns an IAsyncEnumerable or IAsyncEnumerator rather than an IEnumerable or IEnumerator, with IAsyncEnumerable consumable in a new await foreach


async Task GetBigResultAsync()
{
    var result = await GetResultAsync();
    if (result &amp;amp;gt; 20) return result;
    else return -1;
}

What if you want to return a set or stream of results. With C# 8.0, you can achieve this


async IAsyncEnumerable GetBigResultsAsync()
{
   await foreach (var result in GetResultsAsync())
   {
      if (result &amp;amp;gt; 20) yield return result;
    }
}

  • Ranges and Indices

This is another new feature of C# 8.0 which allows the developers to add a type Index, which can be used for indexing. We can create an int index that would count from the beginning or we can also add ^ operator that would count from the end.


Index i1 = 2; // number 2 from the beginning
Index i2 = ^3; // number 3 from the end

int[] a = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
Console.WriteLine($"{a[i1]}, {a[i2]}"); // "2, 7"

C# 8.0 is also coming with a Range type, that consists of 2 indexes – start and end. The expression can be written as x..y


var slice = a[i1..i2]; // { 2, 3, 4, 5, 6 }

Dependencies: Async streams, indexers and ranges all rely on new framework types that will be part of .NET Standard 2.1. The types required to use these features won’t be available if you target C# 8.0 to .NET Framework 4.8.

There are some others active features in proposal and in progress state as well. We can only wait for the final version to be released to see all the features.