Software Architect Rahul Basu: “Without High-Quality Data, Even The Best Analytics or Machine Learning Models Fail”

Rahul Basu is a former Dealroom.co Senior Software Engineer between April 2020 to December 2021 in Amsterdam. During his time there, he was in charge of software architecture that handled insight across 2 million startups and scale-ups, 286k funding rounds, 80k exits, and more than 1,500 unicorns.

He established strategies in using containerisation using Docker and Kubernetes, created 100 real-time ecosystem platforms, and led a team of developers to deliver product features that saw high revenue growth. Let’s take a glance at his career path and learn his takeaways in revolutionising the way data-driven products enable the venture capital ecosystem.

How Would You Describe Your Key Responsibilities And Achievements At Dealroom?

I was a Dealroom Senior Software Engineer, but soon took on a leadership role in steering the software architecture.

My day-to-day was more or less architecting scalable systems to accommodate our high-growing data sets with more than 2 million startups, 286k funding rounds, 80k exits, and more than 1,500 unicorns. I introduced contemporary DevOps practices using Docker and Kubernetes, making it possible for us to release and refresh a hundred ecosystem platforms in real time.

This architectural shift not just scaled up performance but also offered our clients live, in-depth details of investors and startups. In addition to that, I was heavily engaged in mentoring my team and ensuring our product met our evolving stakeholders’ needs.

Dealroom Aggregates A Massive Quantity Of Startup Data. Some Of The Most Technical Challenges You Faced In Architecting Such A Platform Included

One of the hardest was processing data at scale without sacrificing high availability or latency. We had to pull in information from web scraping, user input and a set of external APIs, and put it all together in a structured, searchable way. Getting data accurate, especially in 2 million+ startup records meant building solid ETL pipes and introducing checks for data quality at every step.

The other challenge was providing optimal query performance for analysis and investor searches, i.e., careful design of databases and indexing, plus efficient caching.

You Introduced Docker And Kubernetes To Deployments In Real-Time. Walk Us Through How That Revolutionised Dealroom’s Operations.

Our deployment process was largely manual and time-consuming previously in embracing Docker and Kubernetes. With every new feature or bugfix, there was a risk of downtime or inconsistency between our different environments.

By microservice-containerising our application and having Kubernetes handle orchestration, we were able to automatically roll over updates and manage hundreds of ecologies in effectively real-time. The transition dramatically increased our release cycles.

It also modularised our infrastructure, such that each service was scalable in isolation to handle demand peaks. Ultimately, our clients benefited from faster updates, more system reliability, and more controlled releases of features.

Coordinating A Team Of Developers Is No Small Feat. What Approaches Did You Use To Hire, Onboard, And Effectively Mentor Them?

My approach began by defining each of our developers’ unique role and responsibilities to hire developers whose strengths would support our technical objectives. In onboarding, new developers would work alongside our more experienced developers to get their hands dirty working our stack of technologies; Docker, Kubernetes, AWS, and our pipes of data instead of learning it in documentation.

Once integrated, regular architectural discussions and regular review of code ensured everyone developed, learned, and remained up to date in high-standard manners. Mentoring was done constantly in week-on-week one-on-ones, in which I would provide constructive criticism, provide best practices, and challenge them to suggest architectural optimizations.

Would You Be Open To Sharing A Few Of The Key Capabilities You Introduced To Partners Like Dealflow, Tech Nation, And European Startups, And How These Increased Revenue?

We developed partner-specific dashboards and analysis tools for partners such as Dealflow, European Startups, and Tech Nation. Some of these included custom searches, ecosystem mapping, and auto trend or high-growing company alarms.

By providing these personalised capabilities, our partners were in a position to determine investment potential earlier, compare their ecosystem to their competitors in their geographies, and check market tendencies in quick time.

As a result, these premium capabilities accounted for around 60% of our revenue during my time there, because they solved highly unique needs of various clients; be it government institutions, accelerators, or company venture arms.

Dealroom Is Predictive Intelligence And Ecosystem Mapping Centered. How Did You Apply Analytics And AI-Based Insights To The Platform?

We started by solidifying our pipeline of data. We ingested various sources of data and scrubbed it to a place of consistency. After that, we developed a layer of analysis using machine learning models such as for signals such as hiring sprees, news sentiment and investors’ demand.

We also introduced graph databases to enable maps of relationships between industries, investors, and startups. This helped us generate real-time insights, such as unicorn potential or determining up-and-coming sectors. Our AI-infused capabilities weren’t mere number crunching; it was making actionable intelligence palatable in a way that investors and ecosystem builders could use it immediately.

From Your Background, In What Ways Have Platforms Such As Dealroom Changed Investors’, Corporations’, And Governments’ Strategies To Discover And Value Startups?

Platforms such as Dealroom, Crunchbase, and PitchBook brought info on startups to everyone’s fingertips, turning the ecosystem of venture capital from gut-feel to evidence-based strategies. Investors use live analysis to discover high-growth potential companies earlier, sometimes even ahead of milestone funding rounds.

Governments and companies use these platforms to monitor regional ecosystems, inform policy, and track innovation trends. Overall, this has hastened deal cycles, dispersed funding to new sectors, and created a more transparent, more connected ecosystem of startups.

In Your Opinion, What Is The Role That Technologies Such As Machine Learning, NLP, And Big Data Integration Will Play In The Future Of Venture Capital Platforms?

They’ll become even more integrated. As more and more data related to startups is produced, human analysts won’t be in a position to filter it manually anymore. Machine learning and NLP will become more skilled at recognising underlying patterns, unearthing new funding prospects, and automating parts of due diligence.

In tandem, large data aggregation based on news, social media, even patent filings is going to enrichen those signals. Combined, these technologies will allow for more accurate prediction of which startups will be successful, making VC platforms a necessity for anyone investing in the innovation economy.

Looking Back Over Your Time At Dealroom, What Was Your Most Satisfying Achievement, And What Did You Learn When It Came To Designing Data-Intensive Products?

The most satisfying achievement was a smooth transition to a wholly containerised infrastructure that was capable of processing heavy loads of data in real time. This was more than a technical infrastructure optimisation; it transformed the way that quick we could innovate.

I learned that when it comes to data-intensive product offerings, scalability, maintainability, and quick iteration cycles are not a nicety; they’re a necessity in order to provide real-time insights. Every deployment or release of a new capability must be automated, tested, and verified, because credibility of the platform is predicated on having accurate actionable data at all times.

Lastly, For Anyone Interested In Designing Large-Scale Startup Intelligence Platforms, What Most Advice Or Takeaways Would You Give From Time At Dealroom?

First, be hyper-focused on data quality. Without high-quality data, even excellent analytics or ML models fail. Second, design for day one scalability, because your set of users and set of data can blow up overnight if you’ve got a product that strikes a chord.

Third, never overlook the strength of DevOps practices: deployment, testing, and monitoring automation enable you to allow your team to be innovators. Lastly, never lose sight of the fact that technology is to support in real-world issues, hence always work in consultation with users. Be sensitive to investors’, government’s, and start-up requirements to make your platform relevant and impactful.