Framework Code Complexity Comparison
medium.com/@taylorotwell/measuring-code-compl...Posted Jan 9, 2017, 3:45 PM UTC
46180
medium.com
Good job on keeping the cyclomatic complexity down.
It would be also interesting to show more metrics, like cohesion, coupling (how much classes influence each other), inheritance depth or methods per class.
Complexity can hide not only between lines, inside one method, which is what this test measures, but also between files - in the structure and the architecture.
23
Not to be taken as a flamebait, but comparing Laravel to complete Symfony components feels a bit unfair. Perhaps just checking the components you use in Laravel would be a more suiting fit for comparison. However I do agree with the outcome, a lot of the Symfony methods are long. But some of those long methods could not be done any other way giving the task they have. And yes that is debatable, one could move code to separate methods just to score lower in the method line length however that should increase the complexity score.
Oh.. and isn't Eloquent more of a Active Record then a Object Relational Mapper?
20
Active Record implementations are considered ORMs.
I compared my complete set of components to theirs, while being honest in the article that we do use HttpFoundation and Console components, though no other components are used heavily throughout the framework. I wanted to get a feel for how I write code vs. how other people write code, and all of the projects measured contain enough code to do that accurately. Further, I wanted to dispel any narrative that Laravel is a hack or poorly written.
9
Active Record implementations are considered ORMs.
Active Record and ORM should be considered as separate patterns. Doctrine = ORM + Data Mapper + Unit of Work + abstraction over SQL (DQL) with it's own AST, with bunch of legacy code (which probably will be removed), Criterias and so on. This is far much complex solution to get abstraction over your storage. So basicly you can't compare them.
I wanted to get a feel for how I write code vs. how other people write code
Please be honest. You can compare this only be checking something with the same (or at least similar) set of features. For example you may compare Laravel's IoC vs PHP-DI or some other container. Or you could compare doctrine/cache vs illuminate/cache. Or validation component. This will give "some" realistic difference. But comparing two very different things by using generalized metrics... This is just marketing crap.
20
I think you're missing the point. Average complexity per method is indicative of a particular "style" of coding. It has nothing to do with the solutions. Very complicated problems can be solved with low method complexity. I'm sorry you find the metrics uncomfortable but that doesn't negate the facts.
Doctrine chooses to solve the database problem one way and I choose another way. Nothing about the approach they chose dictated what the average complexity of their methods should be. They chose that, and that is what I'm measuring. If you want to simply throw your hands up and say their problems can't be solved cleanly and with low complexity then that is your decision I guess.
0†
Average complexity per method is indicative of a particular "style" of coding.
Nope, this just means that this code contains multiple execution paths. Nothing more.
Very complicated problems can be solved with low method complexity.
Implement any parser (json for example) with low cyclomatic complexity.
They chose that, and that is what I'm measuring.
What I'm trying to say is that focusing on metrics doesn't solve anyone's problems.
12
Further, I wanted to dispel any narrative that Laravel is a hack or poorly written.
You know very well people are complaining about Laravel's architecture (which it imposes on its applications), which is everything outside the method implementations and not about the quality of the code in the methods.
So you should measure very different things, like public interface complexity, presence of God classes, separation of responsibilities, modularity, leaky abstractions and so on. Not how many lines or nesting levels are there in a method.
8
Laravel doesn't impose any architecture. If you believe it does, prove it.
0†
Oh, yeah, I'm absolutely looking forward to an endless debate, where most of my arguments are answered like "you don't have to do what the Laravel documentation does", and "you don't have to use the components that Laravel comes with", because of course that's why we read docs and download frameworks. So we don't use them afterwards.
Or if that fails "no, this is obviously good architecture, because I have a few big sites written in it and it works fine". Another evergreen response.
Honestly, I would bother if I knew I'm somehow breaking the news to you where Laravel has issues, but I'm not. You know the large issues very well from thousands other conversations you've had here and and elsewhere online. You're just doing this to "win", and frankly I don't know why you're wasting your time, when you neither care to act on feedback, or learn from it.
6
But some of those long methods could not be done any other way giving the task they have
Split it up into sub tasks? I mean I don't have the code base infront of me and he didn't state which specific methods, so I couldn't give you an educated guess, but based on how I see things in Laravel, you could create sub tasks. or refactor the logic down based on the evolving door that is the web.
1
There is two limits to this:
- it's sometimes easier to have a long readable method than having to check 10 different methods
- there is a performance cost. At a framework/library level like Symfony Container/Router or Doctrine, even a simple loop or an additional method call can be a big deal
4
This makes sense, thanks for clarifying
1
This probably won't be popular, but I don't personally judge code with a million small methods as being less complex as code with a smaller number of larger methods. It really depends on how the code relates, and for me it's often easier to follow a medium sized method than follow around a bunch of function calls that are split purely to remain extremely short.
18
Can be. That's why I read through every class and method by hand to look at that sort of thing.
6
Good job keeping complexity metrics down in Laravel, really impressive actually.
17
Numbers are a funny thing. If one wants to assign meaning to these results, it looks like Laravel 5.4 complexity is similar to Laravel 4.0 complexity, despite roughly doubling in size:
https://gist.github.com/pmjones/20109b503a4636fc58046382e7dece75
(Maximum method complexity has doubled, but average method complexity is the same.)
I opine that any complaints about perceived complexity, whether based on early versions of Laravel or later ones, don't appear to be reflected in a numeric comparison.
Of course, if one doesn't assign meaning to the numeric comparisons, then it's meaningless. ;-)
2
Interesting, thanks for posting! Since it wasn't included in the benchmarks, I ran it for Opulence, too. Here are the relevant stats:
Lines of code: 35,796
Longest method: 44 lines of code
Average method complexity: 1.80
Maximum method complexity: 23.00
Percentage of methods that are non-static: 93.58%
15
interesting, first I have heard of this. Taking a look :)
2
I was pleased to see Laravel has the lowest average method complexity of any of the frameworks measured.
And it more than compensates for it by having classes with over 50, 60, I think in one case 100+, public methods.
Too bad "cyclomatic complexity" doesn't measure this, but no matter. You can measure only what you're better at.
Reminds me of when Zune marketing was selling it as "the most well sold hard drive based MP3 player", because at the time competitors like iPod have moved to superior tech, such as flash memory.
14
Compare the two on code climate. Laravel still scores better. I'm sorry this is so troubling to you :)
-2†
I don't know why you're going around this thread and telling people they are "troubled" or what did you say in that other comment... "distressed"... over your marketing shenanigans.
I mean when someone is reading their favorite subreddit for news and occasionally stumble onto your naive arrogance, it may provoke some of us to call you out on it. You're the Donald Trump of PHP framework makers.
The fact you get feedback doesn't mean any one of us is shitting their pants and crying on their keyboards. It simply means we're not buying your B.S.
13
Laravel only uses http-foundation and console in any meaningful way.
Want to let the Symfony guys know to update their page? http://symfony.com/projects/laravel or are you still using all that?
11
Symfony (Only Components, No Twig, No Doctrine)
Would it not be fair to only test the FrameworkBundle instead if you are comparing frameworks only?
Update:
When doing some quick test on these bundles which are most commonly used I guess:
- FrameworkBundle
- SecurityBundle
- TwigBundle
- FrameworkExtraBundle
Then these are the results:
Lines of code: 39,688
Longest method: 58 lines of code
Average method complexity: 1.96
Maximum method complexity: 30
Percentage of methods that are non-static: 98.08%
This is a far more reliable comparison between Symfony Framework and Laravel if you ask me. I couldn't care less about the actual numbers, I just want to point out the article it self is useless.
Just because it's fun the same numbers for Silex:
Lines of code: 5,332
Longest method: 11 lines of code
Average method complexity: 1.82
Maximum method complexity: 13
Percentage of methods that are non-static: 94.59%
These numbers seem "better" than Laravel, was this the reason it wasn't added to the article maybe? :D
11
Complexity is still higher in both cases and absolutely apples and oranges comparison because that is just one small part of Symfony compared against the entirety of Laravel.
1
So Laravel does not have to include it's dependencies for these tests and Symfony and Silex do?
4
I didn't include Symfony's dependencies. If you are having this much trouble understanding I can't help you. I compared Symfony's components to Laravel's components. (Illuminate organization on GitHub)
0†
Symfony Components are not the same as the Framework which uses a subset of those components just like Laravel. For the latter you decided just not to include it in the comparison. It doesn't make sense at all.
5
Symfony requires you to have a deeper understanding of the framework, which can be somewhat daunting. The upside, though, is that a good understanding of the framework and it's underlying architecture will enable you to write better code and have more control over what's happening and why.
1
No it won't. Provide one actual reason why Symfony will let you write better code.
1
It is very important to note that both frameworks are composed from decoupled modules which you can interchange and use with any or no framework. Laravel is mentioned in other answers. It's a really nice framework with an impressively ease learning curve. I would surely use it when building a website or a small app/service. It's neither good nor meant to be good for large scale, complex or enterprise apps, so it really depends on the project. While trying out ZF2, the learning curve was hard. Configuration over convention seemed to make everything an order of magnitude harder than I think it should ever be. Also, it's components felt much less cohesive than any other framework I worked with. I have to say that I probably haven't given ZF2 enough hours to possibly get to love it, but somehow everything I did with it felt as a suboptimal design and edgy architectural decision. Although, it is a great framework with a great company behind it. Symfony wouldn't spare you any time with it's learning curve either. However, after understanding how it's components work in depth, I keep getting impressed day after day. Nowdays, if you use Composer for dependency manager, there is a great chance that you are already using some of Symfony's components, directly or through another library. I'm impressed with: Great dependency-injection container. Configuration through annotations, php, or xml, or any combination. Asset management through Assetic. Incredibly powerful Form component. Easy to use and powerful Event system. Security features and and implementation of the voter design pattern. Native integration with Doctrine 2. Another thing that I want to point out is that the Symfony is the only framework that got a 5 milllion dollar investment (SensioLabs raises 5 million euros to boost the Symfony ecosystem) so you can be sure it will be kept rock solid and continually improved. Also, the community is friendly. Even core developers seem to be very accessible, and are very active in the open source community, which is a pretty valuable thing. Starting with Symfony was as tough as starting with Zend, and they are both great, you can't make a wrong or bad choice with either. However, the first one left an impression of being one of the best things that happened to PHP, the latter didn't. Source: Quora. I would solely depend on your needs the choice of frame work
1
So what is this metric actually good for? I can't really see the point..more complex frameworks have more complex metrics..thats why Laravel is at the level of a microframework? Or is it just about best practise to have short methods?
10
Laravel has more features than Symfony. Queues, command bus, realtime sockets / event broadcasting, etc... that's the point. It has those features but still has lower average method complexity than Slim.
-1†
It has those features but still has lower average method complexity than Slim.
Wait a minute. You just told elsewhere that "features" and complexity are in no way related. Now you say "It has those features but still has lower average method complexity", implying that they are related.
Hilarious.
8
I simply mean that I'm glad it maintains that complexity across a large code base. Not because the feature set is large per se but because it's hard to maintain discipline across a code base of that size with many contributors without "fudging" and letting poor quality code into the code base.
4
Good point, and good efforts in maintaining the framework. I personally think that as the size of a project grows, the stuff you have to compromise in order to keep both complexity and readability in control keeps on increasing.
Sometimes, its better to lean a bit more to the readability side since its getting important day by day. No one's going to make a fuss about a few functions or classes written extra, or that you created a wrapper class for something that could be achieved without it, thus making the code "more complex". But forget to indent code at right places or have variable names that are difficult to understand and people will complain a lot.
2
it's hard to maintain discipline across a code base of that size with many contributors...
So now you "simply mean" that they are related. Got it.
-6†
Apples and oranges, Symfony does not advertise itself as a unicorn which can do everything out of the box. Instead it tries to keep the framework itself lean and let's you choose your own.
6
So does Laravel.
3
The difference is that Laravel ships all these features while Symfony does not. Laravel is heavily opionated on basically everything. I'm not saying it's bad, I'm just not a fan of it.
3
Which version of the Symfony Components did you use?
I think that part of the complexity that is found inside those metrics is due to the fact that Symfony has a really strong BC promise, that forces them to litter the code with an astounding quantity of if (..) trigger_errorfor deprecation notices, which will elevate the complexity by a lot.
7
Latest Symfony 3.
2
Would be interested to see the difference between 2.8 and 3.0 or 3.4 and 4.0. why? Because the latest version (3.2) already contains 2 minor versions of deprecations.
4
I agree. I asked this for this exact reason. You should try to analyze the 3.0.0 tag, that should be the one with the fewer deprecation notices
2
Would love to see the repo that you used to create the benchmarks. Does this include the vendor folder? As although it might not necessarily be Laravel, if the Laravel components are reliant on a zend/symfony components that would in turn would result in a higher cyclomatic complexity.
5
No, no measurements include the vendor folder for any of the projects. I wanted to compare the quality of the code written by the author's of the projects, and I think average method complexity across the frameworks gives a good feel for that.
The main components used by Laravel are HttpFoundation and Console. No other Symfony components are heavily used throughout the framework at this time, nor are any third party components heavily used to build other heavy aspects of the framework such as the ORM, queue, validation, view, templating, etc. libraries.
6
I wanted to compare the quality of the code written by the author's of the projects
Then the title is misleading. Right now it implies that the complexity-metrics say something about the overall "solution" a developer might pick.
I could write my own "framework" which is nothing but a simple wrapper (with all the heavy lifting done by vendor/ code) and get meaninglessly awesome scores on this metric.
9
Except that's not what Laravel is. The ORM, queue, templating, validation, authentication, session, cache, etc. libraries are all first party. And many more. Note Symfony's scores don't even include an ORM or templating layer.
4
Just like Silex and Symfony Framework which weren't tested/published :-)
1
and I think average method complexity across the frameworks gives a good feel for that.
In most of the cases - yes. But not always. Even if it is good to have lower level of method complexity, this isn't a a case when we are speaking about infrastructure code. You can write very complex stuff (like this one) in order to do something in very efficient way (both from performance point of view and productivity, since in may require much less code to be written). Take a look at symfony/yaml parser. It has very high level of method complexity, but overall complexity of component is much more lower than if developers would write some kind of state machine using objects.
Or another example. Will this code be "simpler" if we will reduce complexity? Probably no.
None of the metrics could be used as "absolute" measurement of quality. Especially if we are using only few of them. For example cyclomatic complexity metric could be used with code coverage metrics in order to product something more like "quality" or maintainability index. But as standalone metric, this could only give you "interesting places" to look when you are doing code review.
7
If you feel these methods are as clean as they should be, that's fine. I don't.
2
This would be really interesting if you will "refactor" this in your way. I really don't know how to simplify this without affecting performance.
7
this isn't a case when we are speaking about infrastructure code.
Exactly. I am baffled at the naivety demonstrated by taylor here. I mean, he should have already know these things, right? Or I am doubting that may be he is just very good at marketing stuff...
Has the Php community outgrown taylor and Laraval. That is the big question, I think.
2†
I simply presented some interesting stats. You are troubled by them so much that it's easier for you to assume I am a complete idiot than accept them for what they are.
I even stated in the blog they aren't an "end all" for code quality. That being said, I'll put Laravel up against any other framework using any code quality metric. I feel quite confident it will hold its own. It already scored higher than Symfony on CodeClimate a few years ago before I cleaned it up significantly.
5
at the naivety demonstrated
I think he know what he is doing
4
Has the Php community outgrown taylor and Laraval. That is the big question, I think.
what???
3
How many core members does Laravel have and how many community contributors?
3
One core member and you can find the contributors on GitHub.
2
That's what I find interesting.
Laravel: Core members: 1, Contributors: 377
Symfony: Core members: 15, Contributors: 1398
There's multiple conclusions you can draw from here, but what I really wonder:
- Is it easier to maintain rules in a code base with 1 core member compared to 15?
- Are there certain design decisions in Symfony that have contributed to the longer/more complex code because there was more expertise and thus decided that it was better to do it the way it was?
I think that some frameworks (such as Symfony) benefit a lot from expertise. They have multiple core developers discussing solutions and giving feedback on pull requests. It's kind of like working on a project by yourself, never getting some real feedback or having reviews done.
I can imagine that in Laravel, a lot of decisions did not have counter arguments, or PRs that get merged because there's nobody of a core team saying, "I think this is a bad idea because XYZ".
I'm not saying this has a negative influence on Laravel, but I have the feeling that with 1 core contributor, there's nobody saying: "stop, this is a bad idea!" or "But what about this?".
9
One core member
Just wondering, why its still at 1 core member?
1
This is kinda cool and all, but I think you should add some more information on why these metrics are important and why it's good to improve them.
4
So less code per method to me implies more methods, what about the amount of private methods and function calls? I know that doctrine and Symfony often try to avoid calling too many functions (methods) to prevent this overhead.
What about run-time executed code vs compile time executed code? I also know that some of the more complex Symfony code is only ran compile time.
I don't think it's fair to match a data mapper vs active record as comparison ;)
4
Are you suggesting that more defining more, smaller methods has a measurable effect on application performance when using Symfony or Laravel?
3
I've heard that tossed around though I can't recall seeing any hard numbers or stats on it. I believe there is a cost to calling a function vs running code inline but I would think that is something that is going to only make a large impact in very specific cases.
I think the same argument applies for both sides of the coin. "We made this method long because it was faster that way" should be backed up in the same way that "I took these 100 lines of code and turned them into 20 functions because it was easier to read" should be backed up. Sadly, I think the latter is more subjective and besides things like Cyclomatic complexity probably hard to get stats on.
3
I like u/utotwel article because it's a kick in the pride of a few people that shouldn't take quality of their code for granted and review it a bit. But otherwise I agree with u/simensen: ultimately those comparison are meaningless:
- Depending of what the problem is, it's not always possible to easily split things in multiple methods without making things overly complex
- There is a performance impact. Whilst I believe it matters not in most cases, there is very good reasons why Doctrine UoW is 3K LoC with little methods: performances are a critical part there
- It is a matter of taste/opinion: some people believe a single big method of 100-200 LoC well written is more readable and easier to follow than 10-20 methods
6
This pretty much. While I'm personally in favor of smaller methods, there's some critical processes in libraries that literally get called 1000s of times, ever micro optimization here is welcome.
2
I know that doctrine and Symfony often try to avoid calling too many functions (methods) to prevent this overhe
That sounds like a micro-optimisation at best?
0
When you are a framework/ORM it can matter. It actually matters a lot, look at Doctrine UoW, they decided not to split it for a reason, a micro-optim in it can mean a 5-10% diff in a real world application using it.
3
Micro-optimizations sounds as if it's irrelevant to performance, but it's how SQLite sped itself up by 50%.
So, it depends. PHP doesn't offer method inlining, which would would help a whole lot so we'd be able to keep our code readable and fast.
2
Ah mang, because you set your IDE to show warnings when methods get longer then 13 lines your framework is absolutely superior in every way possible.
Switching to laravel boys!
4†
This will only be a fair comparison if you also include the dependencies on other components. As Laravel uses a lot of Symfony components, it is not fair to just ignore that.
4
These are like stats in sport. Yes they show true facts than can be used to draw an opinion; but at the end of the day it's all subjective and in-reality stats mean fuck all.
:)
3
I'm a sucker for numeric comparisons. Which Laravel repo did you run phploc against?
3
1
Nice. Do you recall which of the other FW repos you compared to?
1
I cloned symfony/symfony but it is a bit of a pain to prepare because the unit tests are nested within each component so you have to go though each one and remove the tests directory. I also cloned zendframework/zendframework and then analyzed only the first-party code.
I would have included Aura as I'm sure it would score well but I wasn't sure how to aggregate it all together.
1
/me nods
Yes, you would need to --exclude-names="*Test.php" or some such -- and I only know that because I'm trying it for Radar now (and later for Aura v2 ;-).
UPDATE For future reference, according to the intent "measure only the author's stats".
$ composer create-project aura/web-project
$ cd web-project
$ rm -rf vendor/composer/ vendor/monolog/ vendor/psr/
$ phploc --names-exclude="*Test.php" .
- Lines of code: 5507
- Longest method: 19 lines of code
- Average method complexity: 1.68
- Maximum method complexity: 9.00
- Percentage of methods that are non-static: 99.78%
(Full results at https://gist.github.com/pmjones/12c6e54d9941c162bedfe57d346ef43e.)
The core Radar ADR libraries:
$ composer require radar/adr 1.x
$ rm -rf vendor/composer/ vendor/container-interop/ vendor/psr/
$ phploc --names-exclude="*Test.php" .
- Lines of code: 4241
- Longest method: 12 lines of code
- Average method complexity: 1.64
- Maximum method complexity: 10.00
- Percentage of methods that are non-static: 97.22%
(Full results at https://gist.github.com/pmjones/8df962c285a127fd26dc7a3aa88082a6.)
3
whatever makes you feel better i guess
2†
[deleted]
2
A more cynical person than me might point out that having the creator of a framework produce metrics that show their framework is superior to others is not much of a surprise.
You're free to run them on your own: https://github.com/sebastianbergmann/phploc
From laravel dependencies:
Basically speaking... this metrics is just meaningless.
Not at all. First, Laravel only uses http-foundation and console in any meaningful way.
Secondly, it very clearly demonstrates the metrics of each framework's first party code, which is what I wanted to measure... how the author's and maintainers of each framework write their code. Again, I'm well aware Symfony developers in particular have a very hard time accepting these metrics, but I simply present them for consideration.
Taylor, what exactly are you using symfony/routing for if it's not a meaningful (or even critical) aspect of routing in Laravel?
Second, of the dependencies listed, the two you mentioned are the two largest and most complex.
I realize you were looking to primarily measure code you wrote which makes sense, as this type of metric is about helping you check for things to improve or give yourself a report card. So, I definitely respect this post for what you present it as, despite the fact that I really (personally) dislike Laravel every single time I have to use it.
I do think that the inclusion of "% static methods" is kind of bullshit though. True, Facades (as you implemented them) are not true static methods that operate without an instance, but the complexity static methods introduce is not about the fact that they're static, it's about the fact that they can be called from any scope and a parent scope cannot restrict a child scope from doing so.
The ability to call something statically even if it's not actually a static method introduces a LOT of complexity and mental overhead in my experience, and basically all of Laravel's complexity, again in my opinion, is hidden away in this little niche. (EDIT: It would add just as much complexity to have a service locator or dependency container that you add a static instance accessor to.)
It also makes writing tests more complex, which discourages testing, reduces ability to reason about code, and reduces stability of the application.
I am not surprised that Laravel scores very well with these metrics, because Laravel's complexity is in places these metrics will miss.
If they're not meaningful why are they still listed as a required dependency?
Oh. Why not then just get rid of the "meaningless" dependencies and code (and maintain) them yourself before shitting on the people you yourself are thus depending on.
And it's really hilariously easy, too, so there's no reason not to.
That's from testing the /src of my own micro framework, took like 2 minutes. Apparently I need to work on the method length and complexity.
[edit] for the record, I ran phploc against laravel/framework and got the following:
Seems legit.
You have been called out by Taylor.. please go ahead and report your results now.
Wow. I don't think I have ever been called out before. Quite a honor. Is there a procedure I should follow? Do I need a really big hat or is that just Hollywood folklore?