Working on a green field project is different from working on an existing one. The latter involves more challenges and constraints. Often, working on existing projects makes us come up with efficient and innovative ways of solving problems.

I usually work on both the front-end and back-ends. I figured out an interesting way of writing the React components while I was working on the internal dashboards. Let me explain it with a basic example.

Let us say we are building a Modal popup component. There are many ways of building the Modal component, but often, the following approach is used

const Modal = ({title, body}) => {
  return (
    <div {...}>
      <div {...}>{title}</div>
      <div {...}>{body}</div>
    </div>
  )
}

<Modal title="My title" body="My modal body" />

This method is precise and works perfectly. You take body as a prop and render that in the <Modal /> component. Need title support? Sure, add another prop title.

You will appreciate this approach until you find the issues with it. Let us say I also want to have a button inside the body I can still do it by passing a React component instead of a string for body. This leads to an argument that when to use children and when not to. I don't want to get deep into this discussion, but certainly, there is a cleaner way of solving this problem. Let us look at the following approach

const Modal = ({children}) => {
  return (
    <div className="modal">
      {children}
    </div>
  )
}

const Modal.Title = ({children}) => {
  return (
    <div className="modal-title">
      {children}
    </div>
  )
}

const Modal.Body = ({children}) => {
  return (
    <div className="modal-body">
      {children}
    </div>
  )
}

# Usage #1
<Modal>
  <Modal.Title>My title</Modal.Title>
  <Modal.Body>My body</Modal.Body>
</Modal>

# Usage #2
<Modal>
  <Modal.Title>My title</Modal.Title>
  <Modal.Body>
    My body
    <button>Submit</button>
  </Modal.Body>
</Modal>

# Usage #3
<Modal>
  <div className="my-own-title">My title</div>
  <Modal.Body>My body</Modal.Body>
</Modal>

The primary difference between approaches #2 and #1 is having low-level components outside the parent component. You can use the children components without parents, they still work as expected, but when you use them together, they solve a particular, bigger, and common problem, in this case, rendering a Modal.

Compound Pattern

I instantly fell in love with this pattern. I used this pattern almost for everything that I built in React unless it is not adding any value. A few advantages of using this pattern are

Have complete control over what goes inside the children.
You can completely replace children with a custom version of the children.
The children can be re-used for another version of a parent or somewhere else altogether.

If you take a step back from React and re-look at the above advantages, you will identify that they are very generic. They don't have to be applied only for front-end apps. This made me curious to find out if people theorized it. Indeed, yes and it is called Compound Pattern. Often it is called Composite Pattern too in the context of React. We build a component by composing other components. It gives more control on what childrens to use.

Backend Table - An opportunity

Moving on, I was working on a particular problem in the back-end. We have multiple dashboards and mongo collections where we dump different types of data. The data that goes inside these collections is huge. The same is the case with the users of these dashboards.

On the dashboards, we have multiple tables which show the records that are there in the Mongo collections ex: The users table. There are multiple controls on UI tables like paginating, filtering, sorts, etc.

Until recently, the front-end used to have a date range filter to show the documents from the collections. Separately, we had a restriction on the date range so that they don't fetch a lot of data at a given point in time. Because every table has its way of fetching the data from the back-end, there was no standard way of loading the data on front-end tables. Besides, because of the date range method of fetching the data, we were either not supporting all the above-mentioned controls on the tables, or there were so many jugaad ways. I refer to this approach as a back-end table/pagination for this article's purpose. We wanted to solve it in a cleaner way.

I believe in "do what works", so, there is absolutely no issue with the above-mentioned approach. I wanted to redo it differently because it was not solving our problem anymore. Implementing back-end pagination is not a complex problem. It is quite common and a solved problem.

What pushed me to write about it is the approach we have taken to solve it. There were a few considerations about the solution

Need to roll it out incrementally for all the collections/tables
It should support pagination, filters, and sorting on every column of the collection
Ability to transform the records before sending the response
Ability to override almost everything (filters, sorting, paginations) because every collection/table has its definitions
We were okay with having a keen eye on indexes of the database

I felt that the Compound Pattern is a perfect fit here. Let me show the implementation of the solution.

def paginate(
      qs: QuerySet,
      sort_by: str = None,
      asc: bool = True,
      offset: int = 0,
      limit: int = 10,
      filters: dict = None,
      projection: List[str] = None
):
    if sort_by:
        qs = qs.order_by(f'{"" if asc else "-"}{sort_by}')
    if filters:
        qs = qs.filter(__raw__=filters)
    if projection:
        qs = qs.only(*projection)
    return qs.skip(offset).limit(limit), qs.count()

The paginate function is a generic one and has no dependency on the API request schema. It just sets offset and limit to the QuerySet. The important point to note is that it returns a QuerySet but not the mongo documents. This lets us update the returned QuerySet if we want; more control.

Similarly, we can add support for filters; a map of key to value, sorting, projection, etc. This function is still independent of the API request schema. We need to somehow get all the inputs of this function from the API request.

Taking a step back, it is a GET /list endpoint to which we have to send back the response. All inputs that are singular values like limit and offset are easy to get from query params. What about filters? It is a map of the key-value pair. We need a way to parse the filters from query parameters. Then we would pass the result of this parse function as input to the paginate function for filters.

def paginate(...):
  ...

def parse_filters(params):
  ...
  return {...}

def handle(req):
  ...

  # we want to apply pagination only if frontends asks for. Backward compatibility
  if req.params.get('paginate'):
    qs, count = paginate(
        qs,
        sort_by=req.params.get('sort_by'),
        asc=bool(int(req.params.get('asc', 1))),
        limit=int(req.params.get('items_per_page', 10)),
        offset=int(req.params.get('offset', 0)),
        filters=parse_filters(req.params) # a separate function
    )
    
    # do something else if you want with qs
    return Response({'records': list(qs), 'count': count})
  
  ...

It gives more control to the consumer. For example, if the consumer wants to apply a filter always, it is easy to do so - {**parse_filters(req), 'vendor': 'Amazon'}. We can come up with different parsers for a particular collection if required. The paginate function itself does not worry about API requests.

Another requirement through the process of development is to support export to CSV. Taking inspiration from functional programming and embracing the Compound Pattern, all we had to do is to have a function export_to_csv(qs) that accepts the QuerySet and returns the list of dictionaries. It again gives us control in deciding what to export, having prechecks, etc. For example, if you don't want to export to CSV if there are more than 1000 records, you can put the checks as explained below.

def paginate(...):
  ...

def parse_filters(params):
  ...
  return {...}

def handle(req):
  ...

  if req.params.get('paginate'):
    qs, count = paginate(...)

    if req.params.get('export'): # control over when to export
      if count > 1000:
        return Response(400)
      return Response({'csv': export_to_csv(qs)}) # control over how to send it
    
    return Response({'records': list(qs), 'count': count})
  
  ...

So far, we just created multiple individual functions to parse the filters from the request, to apply pagination (and other stuff) on the QuerySet, and to export the records to CSV. We can use these functions individually, but if used together they solve the bigger problem: the back-end pagination.

Alternatively, we could just have a function paginate(req, export) that takes the request object and export boolean. Internally, it would still have the above individual functions but in a more dependent manner, I would say. This makes it less customizable which is very much required for any non-green field project. This is explained in the following example.

# approach 1 - no customization
def handle(req):
  return paginate(req)

# approach 2 - still no customization
def handle(req):
  if req.params.get('paginate'):
    return paginate(req)

Instead, taking inspiration from functional programming and Compound Pattern we solved the problem in a better way already.

Summary

You can solve a software problem either by having a single function or multiple, independent, smaller functions. It is a spectrum than just a binary of options. Toward the left (single function) would be less code to write but less control. Towards the right (independent, smaller functions) would be comparatively more code to write but more control. There is no one solution for all problems. It is always better to know multiple ways of solving the problem so that you chose the most relevant one.

It will be equally interesting to talk about the front-end part of back-end tables. Maybe for a future post. Cheers :)