Revenge of the Pragmatists - Baishampayan Ghose
BG has been a part of the Clojure community for a long time. There is evidence on the Clojure mailing list that he had the first Clojure code in production back in 2008! Since then, BG has built a team of 80 Clojure programmers at Helpshift (he's the Co-founder and Ex-CTO). I asked him if he could talk about the reality of building a company on Clojure--the good, the bad, and the ugly of the business-side of things.
Slides
Video
Please forgive the poor audio quality. We had technical trouble with the audio recording and had to use the backup audio. Please use the captions (CC) in the video player.
Transcript
Eric: [00:01] So our first speaker is a frontiersman, he cuts trails, he fends off wild geese, he crossed raging rivers and braved freezing winters, he cleared and plowed virgin land, he built log cabins with his bare hands. Metaphorically, of course.
[00:27] [laughter]
Eric: [00:29] Production Clojure and basing a company on it is pretty comfortable now, but someone had to blaze those trails, it was a vast wilderness, and we're glad he did it. Clojure is a nice language to do this stuff in now, but imagine back in 2008!
[00:51] Please, welcome our first speaker, Baishampayan Ghose, also known as BG. He co-founded Helpshift where he built a team of 80 Clojure programmers. Give him a...
[01:05] [applause]
Baishampayan Ghose: [01:05] Good morning, everyone, again. Before I get started, the title of my talk is, "Revenge of the Pragmatists." That's what I wanted to host...Setting out on, I wanted to take my revenge on the industry with some practical and pragmatic stuff which, as you know, is not really found in our industry.
[01:31] Anyway, so that's some info about me and I'll get started. Who the hell am I? As Eric mentioned, I got started with Clojure professionally in 2008, 2009, when I co-founded one of the first Clojure startups in the world. We were probably the third or fourth company to take Clojure to production.
[01:58] Take a couple of steps back. Before I got started with Clojure, my personal experience was mostly with Common Lisp and Python. By the time I had already built a air travel booking system and a sports-based social network, in Python and Common Lisp, for different companies.
[02:25] I had learned to appreciate the power of Lisp, the beauty of functional programming. At the same time, I was bitten multiple times by mutable state. I was doing [inaudible] in my career, but at the same time, I was extremely dissatisfied with the state of the tools that we had at our disposal as programmers.
[02:53] Of course, also, who are all trained of other industries. I thought that I would do something different. Maybe, build a totally new company and show the whole world that functional programming is practical and Lisps can be used for building real-world systems.
[03:20] That's what I got started with. Before that, there were some obstacles. Let me talk a little bit about nature of business software. If you really look at the business software industry -- I'm sure all of you are pretty well acquainted with it. Ultimately business software is basically an automation of a business process.
[03:44] We're not really trying to invent something or do something really innovative. The most value in businesses is had by automating already understood business processes to software. Of course, those kind of problems are very heavily data-processing oriented, so you take data from random places, process it, and then send it off.
[04:13] Potentially, it's a high throughput a system so you will have lots of users, but it's not necessarily a real-time system. Then too, with high throughput, not many companies get to the stage where they need to even deal with high throughput traffic.
[04:33] Of course, you will be dealing with multiple database technologies. You'll deal with, probably, the traditional RDBMS, and maybe the new noSQL stuff, maybe some [inaudible] databases, some [inaudible] databases. You would have to work with all these different data sources, and pull the data, and do a lot of processing.
[04:53] It's most probably cloud-deployed. We don't really deliver software to desktops anymore, as much, these days. Most written software is delivered to the cloud. Requirements are constantly changing. It's not that, "OK. We need to write a compiler for this business language." It's not that.
[05:15] It's always some fuzzy sort of business requirements which keep on changing every week. Of course, it's all held together with duct tape. We should really look at serious software like the software that IRS uses. I heard that it's written in Assembly language.
[05:37] Just imagine, millions of people are using software which is written in Assembly language, which is unmaintainable and very hard to upgrade. That's the status quo. That was the situation.
[05:57] For that kind of a problem domain, I needed a different language. I needed a language...Of course, I needed Clojure but at that time I didn't know. I thought that I needed a language with Lisp-like semantics.
[06:13] I want my language to be dynamically tight. The static typing, I feel, always comes in the way of experimentation with data, especially. Also, static typing type languages can't really offer the really first class type of experience which I had learned to rely on.
[06:33] I needed first-class functions, of course. Who can do without first-class functions? You need a good library, good system, definitely. You don't need 20 libraries for every problem, but you need at least one or two of them.
[06:48] Immutable data structures. From my prior experiences, I have realized that I would not seriously consider any programming language where mutable data structures are that important.
[07:02] Because even though you as a programmer have a lot of discipline and you write codes the right way and avoid mutation, you can't guarantee the same thing about your coworkers or some third-party software that you pull in as a library.
[07:20] At the end of the day, I really wanted the language to get out of my way. I wanted what should help me solve problems. I use the term, "software development at the speed of thought." That's what I want to do. If I have figured out in my mind a conceptual model of the problem and the solution, going from that conceptual model to executable code, that path should be really short.
[07:51] I should not, or, I need not deal with bureaucracy in a language -- classes, interfaces, and all these design patterns, stuff like that. I was looking, at that time, and then I discovered Clojure.
[08:09] Mind you, at that time I was mostly Lisp-weaning, as they say. I looked at the chain Clojure [inaudible] . I said, "Oh. It's got something to do with Java. It seems to be yet another Lisp dialect. Who cares?" I [inaudible]. [laughs]
[08:27] Later, in a few months, I looked at it again. I realized some amazing stuff. Clojure, as a language, seemed to be extremely well-designed and well thought of. It also seemed to be designed by a practitioner like me who is not scared of reading papers. That was very comforting.
[08:51] I read the whole compiler source code, I remember, back then. I tried to understand what's going on there. I read the source code and I understood it. That was a light bulb moment for me because I have never looked at the compiler of Python, or Common Lisp, or any other compiler for that matter, because those compilers or runtimes are not as accessible as Clojure is.
[09:23] The community, that time the channel probably had 50 people in it. Even then, the community was a very warm and welcoming community. It was full of people who were really experienced. They were far more experienced than me. They were looking at Clojure. That also gave me a lot of motivation, and comfort, and confidence.
[09:51] Also, Clojure is a Lisp. I love the Common Lisp, I love the scheme, but were challenges, especially with Common Lisp, the mutable data structures. Also, the language itself is so complex that sometimes you think of Lisp being a problem, and not really an enabler.
[10:17] It seemed to me that it had all the best qualities of Lisp, at least most of the bad ones were not there. It was fairly well-thought out, very clearly designed...and the JVM, I learned to appreciate the JVM far better after looking at Clojure, because then I was able to look at the JVM without Java.
[10:40] It's very important for us to consider the JVM as a platform, and not just a programming language. I was happy, and I really liked Clojure, and I thought that we should do it. There were still some risks, it was a risky bet even then.
[11:01] It was less than one-year-old, and anything could happen. The community could implode over time, or the creator of the language may take some strange decisions which may break the language, or it may become proprietary overnight. There were so many challenges.
[11:27] There was no tooling. I remember writing Ant build-scripts by hand, because again, I did not have any serious Java experience. I was not really experienced about the Java ecosystem. I was not an Eclipse user, I've been an Emacs user all my life.
[11:45] I didn't know how to generate XML out of it, if there are any buttons that you could click to generate that build-script, so I wrote it by hand, by reading the documentation. That's how we got started building Clojure.
[12:00] Of course, Clojure, back then, did not have all the powerful features that we take for granted today. It didn't have protocols back then. All the cool things that have come out in the last few years, none of them were there.
[12:18] However, I felt even then that the language was complete. I did not feel that this language was lacking any critical feature, or something that would impede me from achieving my goals. I felt that, even if something is not there, I could write it myself.
[12:36] Yes, definitely we did reinvent a lot of libraries that we know about today, in the open source Clojure world, internally, because our stuff predated those things.
[12:48] We have an internal implementation of Prismatic's schema, we wrote it two years before the schema came, we had implementations of our own drivers of talking to databases, we even contributed to lots of open source Clojure libraries because of that. We had to fix so many bugs in so many tooling libraries, we had to fix issues with database drivers, and stuff like that.
[13:20] Of course, a lot of people said that, "It's a Lisp, so it's very risky. Lisp is only used by scientists [laughs] and academic people, it's not used in production," and all these kinds of stuff.
[13:33] [laughter]
Baishampayan: [13:33] I did not have any problems with that because I had already built a production system in Lisp, so no one could give me that spiel, and I would not accept it. [laughs]
[13:44] I was completely convinced about going forward and investing in Clojure, but there was a bigger problem. The bigger problem is this, that how do you convince business owners? How do you convince your CEO to use Clojure, because as the CTO, you are optimizing for certain constraints, but as the business head, the CEO is optimizing for some other, entirely different constraints?
[14:21] In this case, it is about risk-mitigation, and not really gaining any leverage through any programming language. No business founder or business owner today is worried about using the best language available to build their software. They're thinking about taking the least risky path. [laughs]
[14:42] Let's be honest, they have different things to worry about. I think, as engineers, we often overlook the cultural and the people aspects of software development. Hence, we think only in terms of just software tools, and sometimes we miss the bigger picture. How do you really convince business owners to use Clojure, or any new technology for that matter?
[15:12] These are some of the things that may work for business owners. It's very easy to hire engineers, but if it's Java or PHP, no one will get fired for picking Java or PHP, because it's apparently easy to hire Java and PHP programmers. I don't know where they are, because I've never met many good Java or PHP programmers myself.
[15:41] [laughter]
Baishampayan: [15:42] or maybe industry reports. There are all these indices that come out every few months, or some radar, white paper, or whatever. If your language comes up on one of those white papers, then OK, then you can say that you can, maybe invest in that language, or it may be, it's ready for prime-time.
[16:05] Blog posts from competition, [laughs] if you see your competition is using some new tool or some other tool, then suddenly, overnight, that tools becomes legit. Overnight, like NodeJS. The moment some competition started using it, [laughs] I also think about using it, because again, the risk is mitigated because my competition is also using it.
[16:31] Hacker News frenzy, if people are talking about this tool or language, or whatever...database, for that matter, on Hacker News, and people are going crazy over that technology, suddenly, that technology, again, becomes acceptable.
[16:47] Clojure didn't have any of those things. We didn't have developers who already knew Clojure, we didn't have any industry reports talking about the power of Clojure. Of course, there were not many articles or blog posts, people are not really talking about it. What really worked, especially in my case, was the authority that I had over building similar systems before.
[17:25] Also, the responsibility that I was willing to shoulder, because I had to, basically, do everything myself because I could not rely on the rely on the recruiter to recruit Clojure programmers because there are none, for example.
[17:39] I could not rely on some trainer to come and train my engineers, I could not rely on someone to write documentation. Everything had to be done by hand, as Eric mentioned, [laughs] so another [inaudible] .
[17:54] Sometimes, even if you want to fix just one thing, you may need to fix the whole thing. Sometimes it also may mean building that whole thing just to fix that one small thing. [laughs] At least in my case, that was what happened.
[18:12] Company has been formed, we have decided to use Clojure, I've got the approval from investors, our CEO, what to do next? You recruit a team, you build a team, but there are challenges there, as well.
[18:35] [laughter]
Baishampayan: [18:35] Who do you recruit, and how? Junior people don't know much, they don't know what they need to know. Mid-level engineers, they may have some ideas, but they just don't know how. The senior people, they're not interested.
[18:53] Again, I'm speaking with a lot of generality here, [laughs] based on my personal experience. Yours can be very different, but that's what I have observed, that as career programmers, we don't really think of ourselves as athletes, we think of ourselves as accountants.
[19:13] Because an athlete would be practicing every day, an athlete would be investing in having the best shoe that's available in the market, an athlete would always think about optimizing their diet, their exercise, even the aerodynamics of their tools.
[19:35] As programmers, we behave like accountants, where we have already learned some rules or archaic technology, we have some domain knowledge, and we just sit on it. We keep doing the same thing over and over again. That, I feel, needs to change in the industry.
[19:58] To get started, what we did was to target the middle-tier of people who had some experience, they had some professional experience. They had seen how the industry works, they have some idea that, "Hey, even if you use static typing doesn't mean you'll not have bugs," for example.
[20:23] Or, "Java may be industrially accepted, but may not be the best tool for everything," or maybe C for that matter, anything. They already had some experience, and all that experience was in the imperative world.
[20:39] A lot of people say that, "Hey, I know five languages. I know Java, I know C++, I know Perl, I know PHP, I know C," and then I told that person that, "Hey, you only know one language," that they are more similar than different.
[20:54] It's like bunch of cards on the freeway. If you look at it from a macro perspective, all the cards are more similar than different. Programming languages also are just like that.
[21:10] We really had to hammer this idea home, that there is an entirely different world where people are thinking about computation in an entirely different way. It may make sense to consider that as an option, as a potential solution.
[21:31] While hiring, what matters first? Degrees, colleges, years of experience with some technology did not matter. That's what you'd read in résumés today that, "Hey, I'm from this college, I did this major, I have 20 years of experience in Ruby on Rails." [laughs] Those things don't really matter. I have 35 years of experience of addition.
[22:02] [laughter]
Baishampayan: [22:06] Does it even mean anything? It doesn't mean much. What really matters is your grasp over fundamentals. Are you able to take addition and derive multiplication out of it, for example? You need to have good fundamentals.
[22:28] Good fundamentals doesn't have to be any language-specific. It may mean just having a good aptitude for abstract concepts and abstract thinking. Passion to do better, that's important, otherwise you'll be in an accountant.
[22:44] Humility to accept feedback, because I think that is another problem that we have in our community, [laughs] that we are not humble enough. We think that we know everything, because we read Hacker News.
[22:59] [laughter]
Baishampayan: [23:01] The world doesn't revolve around Hacker News, [laughs] it doesn't...and patience to keep practicing.
[23:12] As I said, it's like being an athlete. We don't practice at all, and that's such a shame. We don't have any ethos or habit of practicing programming every day. We just go and do our jobs and perform. It's like saying that, "I'm chilling all the time, just call me when Olympics starts, and I'll go and run."
[23:39] [laughter]
Baishampayan: [23:41] You're not going to win. You may run, but you won't win.
[23:47] Culture fit, I think we often overestimate technical fit and underestimate cultural fit. What do I even mean by that? In a company, when you are building something together, some people will have to work together. For that, there will have to be some overarching ideas which will unite everything. That is, you can say, is a culture.
[24:16] If someone does not subscribe to that world-view, then it doesn't matter how skilled or talented this person is, there will be problems in the future. Culture fit is really important.
[24:33] The last one is more of a challenge than a point to discuss. This kind of recruiting works really well when you do it in a sustained manner. You are hiring people whenever you need them. If I met five people today, and they're all awesome, I'll maybe hire all of them, because I don't know if I'll get them later. That's something you can do.
[24:59] Maybe you're hiring just one person a month, or maybe few months you will not hire anyone. That's also OK. When you're growing company, you're really small, I think a sustained hiring thing works really well. At least it worked out well in my case.
[25:17] Once you are grown, you can't do that kind of hiring, you have to do hiring in burst mode. Someone will come and say, "We need 20 Clojure programmers tomorrow," then what do you do? That becomes a challenge.
[25:30] I think it's much less of a challenge now because hey, we are having a Clojure conference here in New Orleans. There are Clojure conferences happening in Europe, in India, all over the world. The Clojure community is growing, and we have a lot more senior developers now in the community.
[25:52] That problem is relatively mitigated, I feel, but still exists, especially when you are looking for good people, and talented and experienced people.
[26:05] Ultimately, it boiled down to this, that you don't really hire teams, you grow them, because we have this mercenary culture in the industry that, it's like everyone is hired gun. They're all like a Wild, Wild West thing, that you have instead of mugshots, you have problems, and that some bounty hunter will solve that problem and they will get the money and then they will go.
[26:41] You don't build teams like that. A team is much more than a bunch of hired hands. A team has connections with each other way beyond just the work. How do you grow a team, then? Once you have assembled a bunch of individuals who you think are capable of being a part of a team, how do you grow them? Training.
[27:13] I talked about fundamentals, if someone doesn't have the right fundamentals, ask them to read the book.
[27:18] [laughter]
Baishampayan: [27:20] It's our book. We are so, I think, grateful today to have the two of the authors of the book here with us. It's amazing. This book changed my life, in 2003, I think. It's been 15 years now. It changed my life because I was doing Emacs at that time, and the Emacs has Emacs Lisp.
[27:46] Somehow, I heard about Scheme, and somehow I chanced upon this book, and it completely blew my mind. It gave me the perspective which I value even today. I think it has shaped not just my career, but even my philosophy around the software. Start with the book.
[28:09] What we did was, we conducted internal workshops, on-boarding, boot camps, and did a lot of mentoring, recommended the book to not only people who were already hired, but people I met randomly in conferences and events, I recommended the book to them. They read the book and then came and joined me six months later. That's how it happened, as well.
[28:39] Someone was Tweeting around, they wanted to learn Common Lisp, and were asking for a Common Lisp book. That guy came to meet me, and I sat with him, and I handed him SICP. He became a Clojure programmer, a professional programmer, he worked with me for four years. It's great.
[29:06] If anyone here has not read or checked out this book, please do yourself a favor and buy a copy on your way out of New Orleans. You can get it signed by Prof. Sussman. [laughs]
[29:19] Conferences and community-building exercises, very important. When we got started, I attended the first few Clojure conjs in the East Coast. The first year of a Clojure conference, I was there. It was incredible to meet people who were starting to learn or try out a new language. [laughs] It was incredible to meet Rich Hickey, David Nolan, and all the old-timers in the Clojure community.
[29:50] Later, we organized a Clojure conference in India, as well. Zach Tellman was there this year to give the keynote. This was the second year in India. We also started a conference on functional programming, so that was in its fourth year this year. That covers all languages, all domains in the functional programming. That really helped a lot, I think, and then book reading groups and meet-ups.
[30:20] Ultimately, what we were trying to do, was to really do the cultivation and the growing, and not the harvesting, because we had to sow the seeds so that we could harvest later. I think, especially when you're getting started, or when a community is really young, you need to make that investment.
[30:45] The Java community doesn't need that today, but maybe it needed it many years ago, and Sun did that stewardship. In case of Clojure also, I think we, as a community, we have to take the charge.
[30:59] That was training, but how do you scale training? You scale training with culture. One way of looking at culture is, it's like a scalable training process with a feedback mechanism. Another way of looking at culture is that, if two people are having a conflict, you go to culture to resolve that conflict.
[31:29] That culture can be around anything, it can be technical culture, non-technical culture, doesn't matter, because conflicts happen in life. What we need to do is to have a clearly-codified guidelines which will become or behave like an arbiter in these situations.
[31:51] Here I'm talking specifically around technical culture, and something that is well-suited for Clojure. We built our culture through repeatable processes, and by applying them with a lot of rigor.
[32:06] Code reviews, the most important thing, because imagine this, there are two Clojure programmers in the company, one person knows a little bit, another person is very new. What do you do? You have to have code reviews in those cases.
[32:21] Let's say, when you are on-boarding new people, to make sure that everyone writes code the right way, the same way, or maybe in a consistent manner, using the right idioms, using the right best practices, code review, I feel, is essential.
[32:41] Upfront design, that's something which is very unique to, I think, the Clojure community or the Clojure language, because it promotes a lot of design before the actual coding process.
[32:59] Generally, programmers are not really used to that style of programming. What programmers usually do these days is, something that I call debugger-driven development, which is, write one line of code and get into the debugger, [laughs] and then just keep inspecting your runtime, and variables, and all of that, and somehow you evolve your software.
[33:24] We can't do that in Clojure. We can, it has the REPL, but doesn't have side-effects and stuff like that, so it's not really easy to do it that way. Also, in Clojure, we put a lot of emphasis on the data. If you know the shape of the data, the you know which functions you can apply on the data to convert it to your required format. That's pretty much all programming is all about.
[33:51] We did a lot of upfront design exercises, we made that a part of our habit. We made sure that no line of code gets written without proper design document that is vetted by the team, so I would say modeled on top of how Clojure itself is developed.
[34:12] Then of course, style guide, linter tools, we have to use those things, because as a team scales, people bring their own tastes, their own style. Sometimes that is acceptable, it's admissible, but sometimes it may add a lot of complexity to your code base, so you don't want that.
[34:31] That needs to be enforced using build pipelines and code reviews, and tools like [inaudible] and all of that, [inaudible] that kind of stuff.
[34:46] Of course, I think the last thing is very important. Back then, we had the problem of not having libraries, today we have a problem of having many libraries for all kinds of stuff, which is a good problem to have also. I feel the middle ground here, in some way, will be to figure out if we can make this process of choosing the right library easier.
[35:15] Internally, inside our company, we just standardized certain libraries, and we wrote our own, as well. Sometimes, some open source library superseded or replaced an internal library, that's also happened, but we had our own internal list of tools that we were allowed to use.
[35:36] That also, I think, helped because a developer could not arbitrarily pull in some random libraries and use it in production. That helped with the consistency part, as well, because as the Clojure language grew, it got a lot of features. We got transducers, we got protocols, we got few other things, so then suddenly, you could write Clojure code in multiple ways.
[36:02] The code library also increased in size. You could use the threading macro or not, for example, or you could use multi methods instead of using protocols. In those cases, I think that's where some mentoring, and guidance, and some guidelines, helped.
[36:27] Ultimately, what we really wanted to do was to make sure that engineers can decide for themselves. For that, you really had to explain the idea behind these features, which is not very hard with Clojure, because with Clojure, most features are very well-thought out, and there is always some rationale behind it.
[36:47] Whenever Clojure would come up with a new feature, we would do internal tick-tocks. We would discuss those features, and we'd also brainstorm about where we could potentially apply this feature, and maybe someone will do some experimentation.
[37:02] We have always stayed far behind the Clojure release schedules. I know, there are some people who run Clojure alphas in production, that speaks volumes of the quality of Clojure, but we have traditionally been three, four years behind, sometimes, Clojure.
[37:22] We were running Clojure 1.6 or I think 1.5 for quite a long time, for few years at least. That didn't cause us any problems either. That also tells you how awesome the Clojure implementation is, because we were using all the new libraries but the language never caused us any problems.
[37:49] That was about culture. Another thing, at scale, all problems become people problems...
[38:03] [applause]
Baishampayan: [38:03] [laughs] ...because ultimately...let's think about this, for a second, let's consider this, we are building software for people, for human beings. The last time I checked, software was still being written by human beings, as well. Human beings doing something for other human beings.
[38:25] Software is just incidental, if you really think about it. It's bridging a gap between two populations of human beings. That definitely or obviously cannot be the main focus. If your problems are people problems, the solutions are not technical solutions, they're also people solutions. You solve them with people.
[38:55] I just wanted to add this line because I wanted to emphasize the fact that, when we talk about culture, we speak mostly about technical culture and not the overarching culture of the organization. I personally feel that, that overarching culture of any organization is more important than the technical culture, because the technical culture will be dictated by the overarching culture of the organization.
[39:28] Some of the challenges we faced...yeah, so I talked about that. Clojure demands upfront design, and that caused a problem with project managers, because they could see that, "Hey, you're coming up with 100 lines of code, so why did that take three days?"
[39:48] Because you can't really justify that, "Hey, I was doing hammock-driven development," [laughs] because this concept of hammock-driven development doesn't even register in many people's minds.
[40:03] What I mean by that is that, Clojure expects you to get away from the computer when you are faced with a problem. When you have a problem, shut down your computer and get to a comfy place where you can think about your problem and the solution.
[40:21] That definitely caused issues. Some issues, not a lot of big issues. It was a perception thing that, "Hey, Clojure was supposed to make it faster [laughs] but you are thinking so much, you are taking so much time in design. Why?" No one asked that question, that once the software was built, it didn't have serious bugs.
[40:46] Sometimes, lack of good or well-tested, production-tested libraries did cause us problems, in the sense that we were never [inaudible] . [laughs] It was not a big hindrance, we could always write anything that we wanted or needed, but it was a matter of time.
[41:08] In case of a startup, time is of real essence, and you want to get something out into the market as soon as possible. Sometimes that may not be possible, because there is no built-in library that you can pull-in, and you need to write it yourselves. That caused problems.
[41:28] Even though Clojure, the language, never ever caused any problems, that we never had any language-related issue -- some people may have had that, but we built this in serious understanding about the semantics of Clojure, so that we didn't have serious problems with the language -- but there were some incidental problems around the runtime, like some of them were around GC.
[41:54] Again, any large JVM software product will have those issues. Generally, sometimes that also happens in Clojure. It may happen if you hold on to the head, so don't do that.
[42:08] [laughter]
Baishampayan: [42:10] Laziness can also sometimes bite you, so please be careful about wrapping lots of lazy functions on top of each other. If you're doing that, maybe sometimes, realize it and that will prevent any stack overflows from happening in production. That happens.
[42:31] The last thing, I think, was a personal challenge for me, was championing change. Change management is not easy at all. What I could do with Clojure, I failed to do with ClojureScript. [laughs]
[42:48] By the time ClojureScript was ready for the prime time, our JavaScript code base had already moved on. We had hundreds and thousands of lines of JavaScript code, and lots of JavaScript programs. For me, it was super-hard to champion a shift to ClojureScript, [laughs] so I did not even go there.
[43:14] I think now, you can get started with Clojure and ClojureScript, we could not do that at that time. That was definitely a challenge. Some lessons learned.
[43:27] [laughter]
Baishampayan: [43:27] Some of these things are well-known already. Small dedicated teams with powerful tools can achieve a lot in a short time. Functional programming does help in building reliable software, it definitely does, there is no question about it. I've had multiple experiences by now, so never had any memory corruption.
[43:53] We had only two kinds of bugs in our software system, only two kinds. One was, some business logic was implemented incorrectly or incompletely, or the third kind happened when we achieved big scale. Those are problems around distributed systems, can't really be solved within Clojure. Maybe it required some discipline, but that's all we can do about it.
[44:23] Most problems, as I said, are around data processing, and functional programming is ideally suited for data processing. Most business applications can be built with Clojure without any problems.
[44:37] Software estimation is really hard. We find it so difficult to figure out how long it's going to take us to build any software. More so now, with micro-services, and distributed systems, and cloud, there are so many constraints and variables that we don't even take into account when we go about designing our software, or doing estimation around the effort. That's definitely a challenge.
[45:05] With all the layers of abstraction that we are building on top of our systems, which are also weakly abstractions, which are also brittle abstractions, it's going to get, I think, harder over time.
[45:19] Last, but not the least, must try ClojureScript, something I regret not doing, but I have done enough hobby programming in ClojureScript to know and understand the value of it. Maybe next time.
[45:34] Some of the things that Clojure made me do in production that generally programmers don't get to do or don't think that they can do, or they don't even know about things, parsers, interpreters, compilers built into a solid business application, imagine that.
[45:52] Instead of a kingdom of nouns and hierarchy of objects, you have actual parsers which are parsing input data, you have compilers which are compiling things to different output, or you have interpreters which are doing it in runtime.
[46:08] Lock-less memory and lock-less concurrency, highly concurrent systems can be built in Clojure without using a single lock because of the abstractions that Clojure provides. Monadic patterns, we discover them in our code, you never use monads.
[46:28] [laughter]
Baishampayan: [46:31] Did not use them, but we discovered those design patterns, and it was great to understand it from that perspective. Immutable data structures everywhere, imagine the beauty of that, so amazing. Also, we were able to wield the power of the JVM, and I think without [laughs] all the headache of Java. I think that was great.
[46:59] Some big wins, so some info about...so was able to assemble an amazing team, 100-plus people, majority of them Clojure programmers. All in India, by the way.
[47:12] We raised $40 million, the tool, the product that we have built, the SDK, is installed on more than two billion devices worldwide. The company went on to having 600 million-plus monthly active users, and our system, last time I checked, was serving 50,000 requests per second.
[47:35] Multi-data-center, multi-tenant SaaS, 400,000 lines of Clojure. The largest repository has around 250k, and we also have 200k lines of JavaScript on top of that, extra. I think that count would've been far less if we were using ClojureScript.
[47:59] Paul Graham in his article, "Revenge of the Nerds," he talked about using Common Lisp to build his company. Clojure is not for nerds, it's for pragmatists. That's why I call this talk, "The Revenge of the Pragmatists." I feel that Clojure is the ideal language for pragmatist programmers with deadlines.
[48:27] I think I have two more minutes.
[48:32] Some parting thoughts, I'll go through them quickly. Building software is unnecessarily difficult, doesn't really have to be that way. As programmers, we should think about it seriously. Distributed systems is also equals to distributed problems. That means better monitoring, better insights into your systems, better metrics, and tooling.
[49:05] Clojure is a language for building processes, do we have a language for building systems? Rich Hickey gave a talk about this a few years ago, we're not really building programs anymore, we're building systems. Clojure is great for building programs, processes, but we, I think, need a language to be able to define a system. [laughs]
[49:33] Developers need to learn ethics, incredibly important. I think we start with programming when we go to college. That's great, but I think we should end with an ethics class, because programmers have more and more power in their hands right now. Without any ethical or moral guideline, it can cause problems.
[50:02] Last, but not the least, I think Clojure will always remain a niche language, it'll probably never be the number one ranked language in the world, but does it really matter? Hey, we are having fun here today! [laughs] Anyway, last line is this, stop coding, we have done enough of coding, let's build software. Thank you.
[50:27] [applause]
Eric: [50:34] All right, thanks, that was a great talk. It's so surprising, that as a startup, you were able to train people, and just consistently build Clojure programmers. How did you do that? Just time-wise, I've worked in startups where you want someone who can just start tomorrow...
Baishampayan: [50:59] Yeah. [laughs]
Eric: [51:00] and there's almost no time to even talk. [laughs] It's just like, "Just get to work, start coding these features."
Baishampayan: [51:10] I think it depends on the kind of company, and startup, and time and space. When we got started, we had an idea about building a company, but we didn't know what to build at that time.
[51:24] We had around six months to a year of time to figure out what to build. While we were figuring out what to build, we were also building, and that happened in Clojure. That helped us bootstrap the community, as well, while we figure out what to do. That, I think, was a lucky coincidence.
Eric: [51:47] You mentioned a cool metaphor about athletes versus accountants, but then you also said that we don't work enough as a team, that we don't build a team. I was wondering if there isn't some conflict between those two?
Baishampayan: [52:05] No, it's a real sport. It's a hurdle, it's like that, so you have the baton. There is teamwork involved, and then you have individual performance, as well.
Eric: [52:15] Do you think companies should pay the...it's like, I think of a sports team, the team pays the athletes to train, and then they perform, like on Sunday night football, or whatever.
Baishampayan: [52:27] Absolutely. I think what we have, the setup that we have, is that we only have competitions [inaudible] , [laughs] and that's all. No training facilities, or no incentive for training. I think programmers should also think about getting agents, maybe that will help.
[52:47] [laughter]
Eric: [52:47] That, again, is more like the free agents, where you could go from team to team. Do you think that programmers should get higher salaries if that happens?
Baishampayan: [53:00] I think we can demand higher salaries as long as we are getting better. If you can show and prove that you have improved, and you have less bugs in your code, why will someone not pay you more? [laughs] I don't see any reason.
Eric: [53:19] I had another question, you mentioned something like, you haven't met that many good Java programmers, or something like that. I know what you mean, it's nothing against Java programmers, it's just there's so many Java programmers that you have this huge curve of, they could be really good, they could be really bad.
[53:44] Do you think hiring Clojure, and basing your company on Clojure, do you think that it gives you a hiring advantage?
Baishampayan: [53:52] It definitely does in some way. It also does limit your pool to people who are interested in learning those things, but it definitely does.
[54:03] That's what Paul Graham also talked about in his article, that it does give you a competitive advantage, because someone else can just come and poach your teammates, but they may still get poached for skills which are transferable, which are...once you learn Clojure, any other language is nothing.
[54:22] [laughter]
Eric: [54:29] This is a really good question...if you could write your names on the cards when you do a question, that would be nice, we could shout you out.
[54:41] People in the Clojure community, people talking about it being very philosophical, and theoretical, and high-level, so do you think that that makes companies scared, that we're just philosophers, and not practical?
Baishampayan: [54:56] Definitely does, makes companies scared, but that's also because it's not really an accepted norm in the industry. If you look at other programming languages, I won't name them, but they don't have that kind of a culture.
[55:09] Those guys are always writing blog posts like crazy. You see a lot of noise coming from a specific cohort, or people who don't care two hoots about reading papers.
[55:19] [laughter]
Baishampayan: [55:19] What do you do about that? I personally feel that as the Clojure community grows -- Clojure anyway has huge influence over other communities, and other communities have also started reading papers now, that's great -- over time, I think it will change.
[55:36] Also, as programmers, we must take ourselves seriously. Just because someone says that you should not read paper, doesn't mean I should not.
Eric: [55:47] This is, again, no name. I would shout it out, but...You talked a lot about difficult problems like distributed systems, and scaling and stuff, do you think some companies don't have a problem that Clojure is good at solving, that they don't need something as powerful as Clojure?
Baishampayan: [56:08] Sure, they need something more powerful, or they need something less powerful. For example, some people may need assembly, and assembly is, potentially, way more powerful than Clojure, in the sense that you can cause more harm...
[56:21] [laughter]
Baishampayan: [56:24] or maybe you're developing a game, and you have very specific requirements around interacting with specific hardware. It's not just about power, I think it's more about the applicability of the language. Clojure is a general-purpose program, so wherever a general-purpose language will fit, Clojure will fit just right.
Eric: [56:45] Great. Related question, have you ever found it hard, with this awesome language that we have, that the programmers get dazzled by it, and they don't focus on the practical when they're, maybe trying to create a distributed system when they don't need one, and that kind of thing?
Baishampayan: [57:01] Yeah, absolutely, it's happening, but I would say that programmers get dazzled by shiny stuff anyways.
[57:06] [laughter]
Baishampayan: [57:07] They would rather get dazzled by something that's really good, than something which is worthless.
Eric: [57:15] Another question, you mentioned design upfront, and how it scares business people because you're not working, I guess, it looks like you're not working, what do you think it is about Clojure that encourages that?
Baishampayan: [57:31] [laughs] When I got started in Clojure, I had no experience with Java. Today, I can say that I understand Java better than most Java programmers. Why is that the case? Because Clojure lays everything bare in front of you for you to understand and then pick out the pieces that you need.
[57:51] I, personally, feel that Clojure forces you to ask these questions, and then there is also, material is always there and that you can go and read. Also, the Clojure community as a whole, and Clojure creator Rich Hickey being a leader, has also I think...
Eric: [58:10] With the Hammock talk.
[58:13] You're saying, it's like Clojure gives you all these -- even on the JVM, which is a monolithic thing -- it gives you the LEGOs of the JVM...
Baishampayan: [58:24] Yes, it does.
Eric: [58:25] so you have to be a designer, you have to want to put it together in a certain way.
[58:32] About that, you talked about people being worried about the design upfront, but then they never consider the cost of bugs later.
Baishampayan: [58:42] Yea, because again, it's a perception problem. People are OK if there are bugs in production, but they want to deploy faster still. I think it's just a matter of perception and it's not a matter of losing resources.
Eric: [59:02] Then there's the bugs you never see, there's no way to compare it. "We avoided a million bugs just by using an Atom, and now..."
Baishampayan: [59:09] Exactly. It's very hard to justify that to a non-technical audience. They will say that, "Hey, 100 bugs I could've written but I have not."
[59:18] [laughter]
Baishampayan: [59:18] It doesn't make sense in their mind, that why would you have written those 100 bugs.
Eric: [59:22] It's the same thing, it's like lines of code, "Look, my solution, I only wrote 20 lines of code and it's solved, whereas this other programmer wrote 300 lines of code, they look like they're working hard."
[59:34] Do you think that that's a problem with the accounting you do in some of the agile practices, where we have velocity, but then the bugs don't count against the velocity of the tickets?
Baishampayan: [59:50] I think agile is anyway dead, it's agility that I think will live on.
Eric: [59:58] Good answer.
Baishampayan: [59:59] [laughs]
Eric: [60:00] Any last questions? No? I can't really see...
Baishampayan: [60:07] Anyway, I'll be available, please...
Eric: [60:10] You'll be here all day.
Baishampayan: [60:11] Yeah, and tomorrow, as well, so please feel free to pick me up. Thank you so much.
[60:17] [applause]