15 Developer Problem Spaces We're Excited About

4 minute read
, ,

15 problems

So much of how we build and deliver software has changed, yet so much has stayed the same.

UNIX, the original operating architecture that underpins our modern stack, was introduced 50 years ago. Linux, the dominant server operating system, was introduced 30 years ago. We rely on virtual machines, introduced 20 years ago. We leverage EC2 to serve and scale, launched 15 years ago.

These aging technologies power a $350B enterprise software market, over $1.5T in global infrastructure, and is expected to grow 3x — 4x over the next decade.

The industry saw an evolution from the 1990s with the introduction of apps, 2000s introduced hosting & web apps, and 2010s introduced the cloud and services. A veritable renaissance at the application and platform layers.

But when it comes to how we deliver and manage software, we haven’t seen exciting progress. We still rely on DevOps (fka sysadmin) to provision, deploy, configure, maintain, triage servers/clusters/environments. We have needed to reinvent the network, security, memory management, storage systems, at multiple layers (Linux, VM, Docker, Kubernetes, etc.) of the stack to meet the increasing demand and scale. The scale of data is growing at a log rate, yet we still rely on ELT pipelines, cron jobs, static scripts, and manual schema mapping to deal with it all.

If the 2010s represented a renaissance for what we can build and deliver, the 2020s will represent a shift to how we build and deliver, with a focused intensity on infrastructure, productivity, and data. We’ve previously written about this as an evolution to an ‘everything as code’ model.

At Venrock, we are excited about several problem spaces across the developer toolchain and infrastructure. Here are (15) problem spaces we’ve been thinking a lot about:

  1. Infrastructure is still too rigid and inflexible; it needs to look more like traffic pattern migrations across a city, less like tenants living in buildings. New clusters should be able to instantly provision & spin up/down based on job queue and workload demands.

  2. Low-er code is more interesting than no code. Code is good; it’s expressive, allows for desired goals and end states. Too much code also opens up for human error, configuration complexity, lack of self-healing.

  3. Deploying any type of deep learning into production if you aren’t Facebook, Apple, Snap, etc. is still very, very hard and expensive; design, train, test, deploy, manage still very brittle.

  4. Engineering productivity is still a vast, unsolved problem. Looking at ‘commit velocity’ is the wrong metric. We need to remove the overhead from an engineer’s day of updating tickets, etc., and design engineering workflows using engineering design principles.

  5. Containers have become too overloaded, heavy, and costly (K8 = complex). Architectures such as unikernels / microkernels have the opportunity to run 100x more VMs/server, at increased performance, lower cost, and improved security.

  6. A replacement of YAML is badly needed. YAML was introduced 20 years ago at XML’s peak popularity, before JSON existed, at a time when declarative configuration was the norm. Today it’s widespread, error-prone, hard to parse, and running our most important automation pipelines. Not surprising, 30% — 50% of all SEV-1 are configuration related.

  7. Provisioning dev, demo, staging, etc. environments is still too cumbersome, riddled with inconsistencies, and disconnected from your CI/CD workflow.

  8. Security and compliance is still an afterthought for developers. Needs to be designed, written, and maintained as code, by developers, not SecOps.

  9. Documentation is still broken; code is expressive, immutable, version-able, structured… why can’t documentation follow suit?

  10. We still overly rely on DevOps as gatekeepers (see ‘2. death of devops engineer’), whether intended or not. We need to go back to DevOps as a way of bringing the building and delivering of software together, not separate, moving every DevOps-owned workflow to an ‘-as code’ service, enabling engineering to own the workflow.

  11. Most projects don’t need complex containerization and orchestration. Tools like Firebase and Heroku are often sufficient and fast but stuck in an old paradigm with significant limitations in the enterprise. A battle-tested, modern approach to the Firebase and Heroku class tools can make today’s application development 10x easier.

  12. Teams still struggle with getting access to data and being ‘data-driven’… DevOps and data eng are bottlenecks; the citizen data engineer needs to be empowered with real tools, through a central hub for accessing ingested data, previously completed workloads/analyses, collaboration on data sets, etc.

  13. The problem with microservices is there are wayyyy to many of them; we need a better control plane to get a better understanding of what they are, how they are performing, when they are involved in an incident, how to keep them all production-ready.

  14. Batch and streaming continue to be two completely different execution engines and architecture. We need a single model focusing on data properties/logic and latency needs, abstracting rather than building on the underlying runtime engine.

  15. Vulnerabilities occur across applications, infrastructure, and configurations. Organizations need a way to identify, visualize, and remediate these issues, often generated by a multitude of scanners.