Build database

In the lab, once we have the input data with the path above as at the beginning of the lab, we will configure AWS Glue and Crawler to run on a schedule once a day. Crawler will scan the path containing the input Parquet file, save it on S3, and then create a database along with accompanying tables. When there is a new version of the report, the data table will be automatically updated.

Amazon Athena helps us access and view the contents of the parquet file via SQL commands. Amazon Athena is a serverless solution that supports executing SQL query commands on large amounts of data. Athena is only charged for the data scanned, unlike traditional database solutions.

The detailed configuration steps so that Amazon Athena can access data files via AWS Glue are as follows:

1. Access AWS Management Console

   • Search AWS Glue
   • Select AWS Glue

2. In the AWS Glue interface

   • Select Crawlers
   • Select Create crawler

3. Configure Crawler, enter Name as Cost_MasterCrawler. Then, select Next

4. Select Add a data source

5. Configure data source

6. Select S3 path

7. Complete the data source configuration.

8. After configuring the data source, select Next

9. For Configure security settings, you enter the role name AWSGlueServiceRole-MasterCrawler and click Next

12. Proceed to add a database

• Select Add database

13. Enter the database name as costmaster. Select Create database

14. Complete creating the database.

15. Add the database successfully and select Next

14. Review and select Create crawler

15. Complete creating the crawler.

16. Select Run crawler

17. It takes about 1 minute to initialize the run crawler.

18. Initialize run crawler

19. Run crawler successfully.

20. Check the Table of AWS Glue. We see the data table monthly_report

21. View detailed information of the data table.