Netflix has released two applications used by the company’s security team to monitor the Web for potential threats.
Just like other major companies, Netflix constantly analyzes online discussions and posts that might present an interest to its security team. For example, hackers could be planning on launching a distributed denial-of-service (DDoS) attack against its servers. If they are discussing their plans on a social media platform or on a forum, the company could learn of the attack and prevent it from causing any damage.
Two of the security-related applications used by Netflix’s security team are Scumblr and Sketchy, which the company released on Monday as open source.
Scumblr, a Web app developed in Ruby on Rails, enables users to search the Internet for content of interest. Its built-in plugins are designed for searches on seven popular websites, including Google, Facebook and Twitter. However, new plugins can easily be created for manual or automatic searches on other sites, the company said.
“Scumblr leverages a gem called Workflowable (which we are also open sourcing) that allows setting up flexible workflows that can be associated with search results,” Andy Hoernecke and Scott Behrens of the Netflix Cloud Security Team explained in a blog post. “These workflows can be customized so that different types of results go through different workflow processes depending on how you want to action them. Workflowable also has a plug-in architecture that allows triggering custom automated actions at each step of the process.”
Scumblr is designed to work together with Sketchy, which takes screenshots and collects text from websites. This can be prove useful because some of the results found by Scumblr could be on malicious sites. By using the two applications together, security analysts can view the content they’re looking for without putting themselves at risk.
“Although a variety of tools and frameworks exist for taking screenshots, we discovered a number of edge cases that made taking reliable screenshots difficult – capturing screenshots from AJAX-heavy sites, cut-off images with virtual X drivers, and SSL and compression issues in the PhantomJS driver for Selenium, to name a few,” Hoernecke and Behrens said.
Sketchy, which can be used through an API, addresses all these issues. The tool is capable of saving HTML, capturing screenshots and scraping text, all of which can be stored locally or in the cloud (AWS S3 bucket).
In order to allow users to scale Sketchy for capturing data from large websites, Netflix developers have relied on the Celery task management system. As for AJAX-heavy websites, PhantomJS with “lazy-rendering” is leveraged to ensure that the content is captured correctly.
Scumblr, Sketchy and Workflowable are available on Netflix’s page on GitHub.