Does Screen Scraping ClinicalTrials.gov Work?. NLM Technical Bulletin. 2025 Jul–Aug

"Some end users and organizations have used screen scraping tools on ClinicalTrials.gov in an attempt to extract data from a single study or obtain data from a group of studies. The cURL command is a popular, open-source command line utility for interacting with servers that can be used to extract data from websites. However, when the cURL command is used to try to access data from a single study on ClinicalTrials.gov, it provides limited results. This limitation happens because the modernized ClinicalTrials.gov is a Single Page Application (SPA). An SPA is a website that has only one HTML page that constantly updates based on user interactions. When a user attempts to extract data from ClinicalTrials.gov using a screen scraping technology, the response for any URL request is not the actual HTML page, but bootstrap javascript code, which is the code used by the web browser to assemble and present a fully functional webpage containing data about the study."