Topic: Application architecture best practices

Hello, I am trying to understand the best way to organize a complex Rails application, but have not found treatment of this subject in any of the six Ruby/Rails books I have purchased, or on-line after hours of googling and forum searches.  So, I thought I would post the question here in hopes of engaging in some dialogue and gaining some insight. 

Here is an outline to facilitate discussion:

1. App is a 5 module complex work flow system.  (e.g., automobile design aid: module 1 - spaceframe, module 2 - drivetrain, module 3 - aerodynamics, module 4 - environmentals, module 5 - controls)

2. Each of these modules are fairly complex and should themselves be implemented as n-step wizards 

3. The modules can either be used in sequence to complete an end-to-end design project (XYZ Concept Car), or they can be used independently to assist in the design of a single subsystem or multiple independent subsystems.  i.e., a project can encompass one or more modules

4. Some modules will produce data that other modules can use to complete their particular function (for example: controls module needs to know about drive train and environmentals)

5. A user can have many projects (i.e., a portfolio)


Questions:

1. Is something like this suitable for Rails development?

2. What's the best way to organize such an application?  1 app? 5 independent Rails applications orchestrated by a sixth app? 1 app with 5 plug-ins?

3. How should controllers be organized?  One controller per module or multiple controllers per module?

4. How should data be organized?  Single or multiple databases?  One model per controller or multiple models per controller?


Thoughts?

Dondi.

Re: Application architecture best practices

1. Rails is ideal for this type of project, since it seems more like a web application than a site.
2. Have you checked out the acts_as_wizard plugin?
3. Controllers should be organized by scopes. IOW they should only handle relevant chunks of data. Many times you will have controllers without models behind them. It's never a strict 1/1 relationship, especially if you have complex login/authorization. It really depends on what you are trying to do.
4. As always this really depends on your needs, but from your description I think this is feasible with just one DB.

Also, for clarification purposes, your concept of module sounds more like a Model. In Rails ... modules are just little helpers you can mixin to your application.

Re: Application architecture best practices

Thanks pimpmaster for pointing out the meaning of "module" in Rails nomenclature.  That's the meaning I intended to convey though -- the term "sub-application" might be a better descriptor.  In otherwords, each of the five "sub-applications" are of broad enough scope and utility to warrant standalone operation. 

As such, one question I am struggling with is whether these should be structured as five independent apps or 1 app with 5 sub-applications.  My bias is towards 1 app, but I am trying to understand if there is a best practice for organizing such complex apps.  i.e., is this why we have Rails plug-ins?

Yes, I have checked out the acts_as_wizard plug-in.  It looks like it might be ideal for workflow management within each "sub-application".  Do you know if it can be used to manage workflow between the 5 sub-applications as well?


Thoughts?

Re: Application architecture best practices

Rails plugins are pretty limited in their scope. They are really meant to add specific chunks of functionality to your application. The type of abstraction or "modules" to which you refer are known in Rails speak  as "engines"

Engines are a pretty controversial idea. Some people swear by them and others find them unwieldy and loaded with incompatible code. Read more about them here:

http://wiki.rubyonrails.org/rails/pages/Rails+Engines

If you ask my opinion, personally I envy what the Django folks have going on under the hood. Basically you have projects and inside those projects you have applications (modules). Its a much cleaner approach IMO and I hope that one day Rails will adopt a similar philosophy.

This is really my only gripe with Rails the framework. In every other respect it has been a dream to work with. Django is cool too and worth looking at, but I fell in love with Ruby so there's no turning back for me.

Re: Application architecture best practices

I'm smitten with Ruby and Rails as well.  After stellar results with my "test project" I am sold on Rails for developing a series of more complex applications along the lines I described in the initial post. 

However, I have yet to see anything that discusses best practices for designing and organizing complex applications with Rails -- most of the apps discussed in books and articles seem to be standalone capabilities such as to-do lists, project management, book stores, music stores, etc. 

In contrast, my app ideas are mostly complex, multi-function, workflow oriented web solutions that would probably be too complex to tackle using lower-level frameworks and development environments.  I'm probably not looking in the right places, but none-the-less I am at a loss.  That's why I posted the topic in this particular forum.  Thanks for your prior validation that such projects are ideal Rails applications.  I really appreciate your input and perspective, but I can't imagine that you and I are the only ones with questions and opinions on this basic issue.

I did look at Rails Engines and did encounter the religious war surrounding the concept.  I came away with the impression that Engines were meant for automating the development of common and often repeated application capabilities (e.g., login, authentication, etc.); however, I will take a fresh look and post my impressions. 

Dondi.

Re: Application architecture best practices

Thanks for the pointer pimpmaster:

pimpmaster wrote:

Rails plugins are pretty limited in their scope. They are really meant to add specific chunks of functionality to your application. The type of abstraction or "modules" to which you refer are known in Rails speak  as "engines"...

After some additional follow-up I found this great article (and Engines development community): http://www.mail-archive.com/engine-deve … 0344.html.  Here is a (selective) quote that speak to exactly what I am looking for:

James Ho wrote:

We have been developing with engines version 1.1.2 for some time.
However, we are going through a major code cleanup. Our development
happened organically, first we built a single rails app, then we
realized we needed to break our efforts up into several rails apps.
Thus, we discovered engines and used them to "peel" out a slice of our
main application so that we may share it amongst our other apps (mainly
models). Later this grew to a few more engines that included the full
MVC compliment.

As we reboot our development efforts in rails 1.2 we are now developing
our applications with engines as a part of the infrastructure from the
ground up. ...Since a plugin does not have it's own generators it would be quite
a maintenance nightmare, not to mention destroying the elegance and
abstraction of engines, to generate them in the top level rails apps and
keep copying everything down to the engine.

I have selectively quoted, so please do not rely on these excerpts to draw conclusions on what the author intended (he was asking a question on how to manage simultaneous development of engines and the apps that use them).  With that disclaimer aside ...

This sounds exactly like what I need for the complex, quasi-self-contained-inter-dependent app ideas that I would like to pursue.  However, there seems to be a major divide in the Rails development community -- you are either against Engines or you swear by them.  Further, it appears that the Rails core developers are allied on the 'against' side.  I don't get it.  It seems like there are a class of (complex) applications (not to mention a market for drop-in applications) that Engines can enable.  Since the core team are people much smarter than me, there must obviously be some major reasons (that escape me) why they oppose Engines.  But, back to my question ...

Engines look like a promising path for my apps.  A few additional questions:

1. Given the controversy, is it 'dangerous' to incorporate Engines into an application?

2. Are there any good articles that outline the Rails/Engines architecture?

3. Any better recommendations than Engines for these types of complex applications?

4. Assuming that Engines is a viable approach, does this overall app structure make sense:

Design_a_car_app (w/ app-level models, views & controllers)
-- spaceframe_engine (w/ engine-specific models, views & controllers)
-- drivetrain_engine (w/ engine-specific models, views & controllers)
-- aerodynamics_engine (w/ engine-specific models, views & controllers)
-- environmentals_engine (w/ engine-specific models, views & controllers)
-- controls_engine (w/ engine-specific models, views & controllers)


Thx.

Re: Application architecture best practices

The engines approach only makes sense for you if you want to use these separate components in other applications. Do you foresee the need to use an aerodynamics engine in another project, or are you trying to make things modular prematurely?

I dont have much experience with engines myself, so hopefully someone else can give you more insight on this.

Personally, if it was me doing this project, I would probably just build it all in one application, then if I saw the need to abstract out some functionality I would make an engine out of the code I'd already written.

In reference to your previous question, I did a little digging and found this great plugin which allows you to work multiple databases. Havent given it a spin yet but it looks promising:

http://www.elctech.com/2007/3/8/using-a … -databases

Before you go gung ho, really think about the implications of this. Multiple databases only make sense if you expect traffic levels of huge enterprise, tens of thousands of hits per second.

It's good to research how far a framework will take you, but realistically think about the intended scope of your project... are ten thousand car engineers going to use this program? Will you be shopping it around to various manufacturers or is it a one-shot deal?

Your business strategy is key in determining what technology you should use.

Re: Application architecture best practices

Thx for the additional pointers pimpmaster. 

I am probably making things modular prematurely.  I do foresee some of these functions being portable to other applications, but this is a secondary priority.  If I were to go the single app route, are there any best practices that would facilitate subsequent engine abstraction/extraction? 

Thanks for the link to the multiple databases plugin.  I don't imagine the app will ever get to the tens of thousands of hits per second scale so I believe a single database should suffice.

Dondi.

Re: Application architecture best practices

I wish I had the experience to answer your questions, but since I haven't written engines before (the whole thing was a little over my head), my advice would be misleading. Hopefully someone else can chime in here, but as you are probably starting to notice, engines have not achieved wide-spread adoption yet.

In your shoes I would hunt down developers who are actively developing engines and ask them directly. One of the things I love about the Rails community is how helpful total strangers are. All it takes is a short email to clear any doubts.

Good Luck with your project!

Re: Application architecture best practices

And do not forget to leave for us a message smile

Re: Application architecture best practices

@pimpmaster: thanks for all of your suggestions.  I'm spending some time with the engine-users mailing list.  I've already learned quite a bit; but I think your prior suggestion was spot on -- I'll concentrate on the app functionality first, then focus on extracting engines later.

@mrorm: thanks for weighing in.  The Working With Rails site is interesting; but I can't quite figure out how to engage it.  Can you break it down for a newbie?

Dondi.

Re: Application architecture best practices

Hi Dondi & Pimpmaster,

I'm not an expert, and I don't know what engines are, but I am thinking about how to organize my controllers and came across this discussion, so I figured I'd put in my two cents.

Rails is an MVC framework.  Let's think about what "C" means.  "C" stands for controllers.  The controller layer is where you collect and organize all of your business rules.  That is, business rules and decision making ability generally DOES NOT leak into your M or V level, but instead these rules are grouped into named controllers. 

So, each controller represents a group of business rules or a decision making process.  Using this line of associative reasoning, your sample workflow application, I suggest, should be organized like this:

/controller/spaceframe_workflow_controller.rb
/controller/drivetrain_workflow_controller.rb
/controller/aerodynamics_workflow_controller.rb
/controller/environmentals_workflow_controller.rb
/controller/controls_workflow_controller.rb

And each step in your workflow should be a different action.  Here, all of the business rules associated with your spaceframe construction workflow are grouped in the spaceframe controller.    And all of the business rules relating to the drivetrain construction workflow are put into the drivetrain controller.  Now what about actions?

So let's say your spaceframe wizard/workflow has three steps.  But what are steps?  Steps are a further "breakdown" of business rules and decisions.  The business rules & decision for each step would be in an action.  Let's say you were hosting your application at yourapp.com.  And you had an action called "step1," "step2," and "step3."

yourapp.com/spaceframe_workflow/step1
yourapp.com/spaceframe_workflow/step2
yourapp.com/spaceframe_workflow/step3
yourapp.com/spaceframe_workflow/finished

You could do the same thing for your "drivetrain" workflow:

yourapp.com/drivetrain_workflow/step1
yourapp.com/drivetrain_workflow/step2
yourapp.com/drivetrain_workflow/step3
yourapp.com/drivetrain_workflow/finished

So basically the mapping for your process is:

WORKFLOW -> CONTROLLER
WORKFLOW STEP -> ACTION

No rocket science here.....I'm just reasoning from first principles: MV*C*. 

Just as CONTROLLERS are about processes, MODELS are about things whose features you what to track.  That's where you would track features of your drive train and space frame.

THING WITH FEATURES YOU WANT TO TRACK ==> TABLE IN DATABASE ==(WRAPPED BY)==>  MODEL

I'm just guessing but maybe there is something called a "space frame" with actually features that you want to track .... like height, width, weight, and price.  Then you should have a table called SPACEFRAMES and a model to wrap that up called /model/spaceframes.rb. 

/model/spaceframes.rb
/model/drivetrain.rb
/model/controls.rb

etc.... The model is where you model your world with THINGS THAT YOU WANT TO TRACK IN THE DATABASE.  That model may or may not correspond in a 1-to-1 fashion with your workflow.  There may be a complicated combination of features with different values of your model objects that need to interact to be able to make a decision in your action/controller/workflow/business rule.  (E.g. If Controls.temperature * Drivetrain.mass > CRITICAL_HEAT redirect_to :action => 'emergency_shutdown') That's why we separate those three layers.

If you wanted to actually edit/destroy/view the values of a spaceframe, then you would do that in the spaceframe controller ((/controllers/spaceframe_controller.rb)) as distinct from the spaceframe workflow controller (/controllers/spaceframe_workflow_controller.rb).

Your URLs for managing your things in your world woud look like this:

yourapp.com/spaceframe/list
yourapp.com/spaceframe/view
yourapp.com/spaceframe/edit

yourapp.com/drivetrain/list
yourapp.com/drivetrain/view
yourapp.com/drivetrain/edit

yourapp.com/control/list
yourapp.com/control/view
yourapp.com/control/edit

etc. 

Does that answer your architecture question?  I think it's a pretty innocent and straightforward way to organize the app as you described it, but it should suffice for the first iteration.  As you guys were saying before, if you need to get fancy and abstract things out and refactor, then you can do that in future iterations as you see necessary.

...


As an afterthought, I guess you can think of controllers and actions as two levels of "directories" to organize your business rules.....hmmm.... what if you need three levels?

Last edited by dbit_solutions (2007-05-14 06:10:47)

Re: Application architecture best practices

When I started out with Rails I struggled a lot with how to split up controllers. It was one of the most difficult things for me. Here's a few tips:

1. Don't split them up based on how they are accessed (public vs. admin)
2. Don't split them up based on how you want the URL to look.
3. Don't split them up based on your site's menu structure.

Instead, split them up based on application behavior. This didn't really click for me until I adopted REST. Think of managing resources and dealing with CRUD, a controller should focus on one resource. Of course not everything fits into CRUD, but this often applies to the majority of the app.

Railscasts - Free Ruby on Rails Screencasts

Re: Application architecture best practices

Hopefully I am not taking us too far off topic...

I built a rather large system recently and most of the design decisions were based around dbit_solutions' feelings on the purpose of the controller within the scheme of MVC.

However, it seems that lately, with the RESTvolution going on within the community, a lot of code is being pushed towards the models with their relationships representing much of the business logic.

Is there any definitive answer to which method is better, or is this just a holy war of sorts where each side "believes" that their method is indeed best practices?

Last edited by raif (2007-05-14 17:41:22)

Re: Application architecture best practices

I've always been a fan of skinny controller, fat model approach. Shoving as much as you can (within reason) into the model makes testing easier and keeps the controllers and views small and simple.

But I see where you're coming from. If you practice REST there's really not much "padding" between the user interface and the database itself. This keeps things simple and consistant, but may be too rigid if you need to keep the view flexible.

At least that's what I thought at first, but now I'm realizing the model and controller layers are still there so you can add the flexibility and "padding" where necessary. Virtual attributes are a perfect example of this.

This kind of goes with the "convention over configuration" idea behind Rails. Initially you should make the database reflect the view directly with little "padding" inbetween (convention) but later on when you need to change the interface and alter the view, you can add this in either the model and/or controller (configuration) if you don't want to change the database structure.

Okay, hopefully that made some bit of sense. wink

Railscasts - Free Ruby on Rails Screencasts

Re: Application architecture best practices

I would like to ammend my comments above.

I was just discussing the architecture of a web application that we are building now with my partner who has about 10 years of experience in programming & software building/design.  I told him my approach above and basically he said (I'm paraphrasing.) "That's an acceptable but not the best way of organizing the code, because it's not modular.  What you'll find is that after refactoring & modularizing your business logic, your business rules will tend be clustered around user actions and informed by your data model." 

I'll paraphrase some more:

Basically, what you need to do is write down a list user actions.  TRY TO FIGURE OUT WHAT THE USERS ARE REALLY TRYING TO DO.  You can do this by writing out in detail the workflow, then extracting out the unique actions that the users are trying to accomplish.  For example if this were eBay you would have two groups of users.  BUYERS and SELLERS.  First I'll write down a list of actions that each user class will engage in.

SELLER:
* Seller signs up
* post item details
* view bids
* accept a bid
* get notified by email that somebody bid on your product
* view & respond to questions on an item

BUYER:
* Buyer signs up
* view item details
* get notified by email that a bid was accepted
* bid on an item
* ask questions on an item


After making that list.  You should, WHILE THINKING ABOUT YOUR DATA MODEL, GROUP THOSE ACTIONS INTO CATEGORIES like this:

SELLER:
* Seller signs up -  USER
* post item details - ITEM
* view bids - BID
* accept a bid - BID
* get notified by email that somebody bid on your product - EMAIL
* view & respond to questions on an item - COMMENT

BUYER:
* Buyer signs up - USER
* view item details - ITEM
* get notified by email that a bid was accepted - EMAIL
* bid on an item - BID
* ask questions on an item - COMMENT

THOSE CATEGORIES WILL END UP BEING YOUR CONTROLLERS.  As you can see that we have extracted out five controllers for our eBay application:

1. USER
2. ITEM
3. BID
4. COMMENT
5. EMAIL

This is a more modular way of defining your controllers compared to the workflow-centered way of designing your app, because business logic will not be repeated over and over throughout the different workflows.  What if you had 100 workflows.  Would you want to encode the email notificaiton business logic 100 times in the 100 different controllers?  Of course not.  That would not be DRY.  In the the eBay case, we have 2 workflows one for buyers and one for sellers.  If we didn't extract out the commonalities between the two workflows we would essentially be doubling our business logic code.

Notice that NOT ALL OF THE CONTROLLERS ARE DATA-CENTRIC.  The controllers: USER, ITEM, BID and COMMENT are obviously data, but the EMAIL notification is more of a group of system actions, for example.    So, while the data model should inform the grouping of the actions they do not fully determine the existence of the controllers.  Some controllers may have actions without being part of the data model.

It's in the views where you will talk to different controllers.  For example, on the view item page in the ITEM view page, you will have a form that links to the BID controller for example.  And on the BID view page, you might have a link to the COMMENT controller so that users could leave and respond to comments.

Last edited by dbit_solutions (2007-05-14 23:14:39)

Re: Application architecture best practices

He gave you good advice dbit. That is also the conclusion I have come up with and it goes right along with REST.

Edit: I should add, for the most part you normally don't need an "email" controller because this is often better handled by observers. For example, you can create an observer to check when a bid is made, and when that happens an email automatically goes out - the controller doesn't even have to be involved.

Last edited by ryanb (2007-05-14 23:15:59)

Railscasts - Free Ruby on Rails Screencasts

Re: Application architecture best practices

I don't know why I didn't come here earlier, what a wealth of information.

I had no idea that rails had implemented the observer design pattern.  I am reading about it now and it will make an excellent addition/upgrade to a lot of my code.

I spent all day yesterday re-reading about the mvc model and REST, to see how the lines are drawn within the architecture of ruby on rails.  I have come to the conclusion that much of the purported differences in design philosophies with regards to separation of the model and controller comes down to a focus issue and how the developer follows through in his execution.

For my next project I will be sticking to the following guidelines.(I would appreciate any feedback!)

1) I am going to follow the REST design in the sense of relegating the controller to simply communicating the actions of the user to the system and concentrate on the composition of my models in an effort to reduce unneeded complexity to simple crud operations.

2) I will however, not attempt to wedge every user action into the REST paradigm as David Hansson sees it.(i.e. everything is a crud operation)

3) I will use observers and filters to handle operations transparent to the user(i.e. authorization and authentication, email notification, etc.)

It is funny because I don't feel like that is much different than what dbit's partner outlined, nor what I did originally.  In my last project though, my focus was a deterred a bit when I let my practices slip and starting relying on the controller as the main housing for any business logic that spanned more than one model.  REST seems like it will be a good direction because by limiting the ease of controller action creation, it will guide my focus towards having the model relationships dictate proper business logic placement.

Re: Application architecture best practices

Thanks guys... this is great stuff.

Re: Application architecture best practices

ryanb wrote:

you can create an observer

I think observers are sweet, but sometimes I get an unsettling error that makes me wonder about their future:

DEPRECATION WARNING: observer is deprecated and will be removed from Rails 2.0  See http://www.rubyonrails.org/deprecation for details.

Of course the site says nothing about observers at all, but I have seen this warning a few times since upgrading to 1.2. Can anyone confirm this?