New Bilara update

For the past several years we’ve been using the Bilara webapp for translation. This was developed in house. We’ve encountered a number of shortcomings, and decided to rebuild from scratch. We’re using a completely different tech stack (elastic/HTMX/apollo/lit). It’s taken, well, a lot longer than I had hoped!

For the past few months I’ve been working steadily on this with Hongda, our main developer. In fact we’re meeting in a few minutes. We’ve made a lot of progress, but I won’t lie, there’s still a lot of work to do. I’d love to be able to open it up for translators to try out, but it’s still not ready.

The idea is to keep all the features of the old Bilara and add a bunch of new ones. I won’t rehash the old features. But here’s some screenshots of new things.

This one shows the “notifications” in the top bar. You can subscribe to a set of translations and then get notified if there are any changes. Most commonly, this will be used for translators working from English, who will then see when the translation is changed.

Here’s the notifications page, a list of translations, and you can click to see the diff.

Settings allow you to select who you get notifications from.

Separately, you can also see various features from the dropdown menu.

Unpublished changes shows your own changes. Normally when you translate, you simply work until you’re ready, then publish. But sometimes you may forget or be unclear about what changes you’ve made. This lets you review them. It takes you to a page that’s similar to the “notifications”.

Here you can see exactly what you’ve changed and make any adjustments.

We also have an Admin panel to help set up and manage users and projects.

User management:

Project management:

And ability to create new root files and change segments. Currently this is all done by working directly with JSON files.

Okay, to the translation page, where most translators will be living. Mostly the layout is familiar to old Bilara users.

But we can do a bunch of new things. Pali lookup!

We have click to search: just click on a word in the root or translation, and it will immediately populate the search column.

Search seems a lot faster, but we’re using it locally so we’ll have to see how it works on the web. Also, we should, I hope, have eliminated the problem with ghost results in search.

Column width can be dragged to whatever you like.

Related projects can be added and removed easily, without refreshing the page.

There’s another specialized feature that’s for those doing primary translation, i.e. working directly with Pali (Brahmali and myself) or Chinese (Charles) or others. You can split or merge segments in the browser.

This is only available for primary translators who are experts in the root language. We’ve built the user management accordingly.

This has two main uses:

  • It makes adding new texts much easier. You can do an initial pass at segmenting the text based on punctuation or whatever, then revise it as you go through the translation.
  • It also helps to fine-tune occasional cases where texts are segmented inconsistently, i.e. where the same passage is treated as one segment in one place, and two segments in another place.

When you split, it’ll tell you to check the HTML, which may have to be manually adjusted. By default, it inserts {}. which is just “no html”.

When splitting, there’s no way for the app to know where you want it to split, so the editor must make the adjustments by hand to the root text. Typically this is done for cases where punctuation and editing are inconsistent in different places.

It’ll add a new segment, and the correct adjustments to the segment numbers will be automatically propagated in the bilara-data files.

Merging is done similarly, except normally merges can be predicted—you simply add one segment to the other.

When splits or merges are made, translators who have translated that segment will get a notification. They can adjust their translation if necessary. Normally this is not really critical.

  • If a segment is split into two segments, the translation is attached to the first segment.
  • If two segments are merged into one, the two segments of the translation are both in the new segment.

But anyway, it’s good to check any changes and make any adjustments.

There’ll be a bunch of other things—between writing the last paragraph and this one, I met with Hongda and he showed me some new features. And there’s still a lot of testing and refinement. But it’s coming along well.

Oh, I’ve also reviewed the color schemes for contrast and accessibility, and added a new one (aruna) just for fun. It’ll be pretty easy if users want to create their own color scheme.

12 Likes

This is incredible! Truly amazing work!
:folded_hands:

6 Likes

Amazing, Bhante! Thank you & Hongda for your work. :slight_smile:

:hugs:

5 Likes

Well done!

I’m sure @HongDa faced quite a learning curve just getting to know the new stack.

I’ll be interested to see how you fare with ElasticSearch. I’m mostly doing maintenance work for the main site, but eventually I’d like to review the search backend for the main site.

That could involve:

  • Refactoring and making improvements to the existing arangosearch based engine.
  • Using Elastic as the backend.
  • Writing something highly customised with Tantivy, a Rust crate inspired by Lucene, the Java library that underpins Elastic.

Plus I’m reading this:

https://www.cambridge.org/highereducation/books/introduction-to-information-retrieval/669D108D20F556C5C30957D63B5AB65C#overview

But yeah, back to tidying our docker configuration at this point.

6 Likes

Welcome to software development. Sounds like you and hongda are doing it right! I look forward to the release.

3 Likes

We’ll be getting in touch to see if you can do some testing!

3 Likes

We literally just got rid of elasticsearch on SC a few years ago!

One thing that’s similar in the two sites: we use the same thing for both site db and search. On SC that’s ArangoDB, on Bilara it’s elastic.

The reason we switched from elastic was so our back end devs could have a single means of querying the data, namely AQL.

I don’t have any real opinions on this, I’m not a back end dev, but the folks at STXnext wanted to use elastic so that’s what we have.

For myself, frankly I feel like I don’t really want to make any major software changes going forward. Enough! I’ll bring the new Bilara to completion, then it’s maintenance and gradual development, but not rewriting from scratch.

The best approach IMHO is to support multiple interfaces, perhaps a specialized search interface or whatever, independent of SC main. Maybe one day it’d replace SC, who knows. But for the foreseeable future, we’ll be simply iterating on what we have.

3 Likes

I see. So the new Bilara uses Elastic as a document store as well as for search?

Well, quite. We can gradually improve the existing design by refactoring. Good test coverage is a prerequisite for this approach.

That’s also what I had in mind. Take the existing code and refactor towards a ports-and-adaptors model where we can plug in our indexing engine of choice. Eventually SC-Search could be a separate service, either running on the same VPS or hosted separately.

Anyway, these changes would be incremental and happen over the course of time.

1 Like

Yes.

Indeed yes. I mean hindsight is 20/20 but I do feel we’ve spent waaaaaay too long on rebuilding.

Sounds neat.

1 Like

We just added a new feature tag. This was because we had an enthusiastic volunteer who wants to tag all the suttas. Tags are a top-level data type. The “tagger” defines a list of tags, with expansion and definition. Then they can go through the texts and add them at a segment level. As many as you like, comma separated (like our “variants”). This is all already built!

I was really stoked to see how quickly this went from idea to actuality!

5 Likes

This will be good fodder for the search index!

1 Like

It might be good to set out some guidelines for tagging. Especially regarding synonyms and English vs. Pali. If tags are at the segment level, will every segment in a sutta related to a topic be tagged? It can get messy very quickly. I have some experience with this on a large-ish Wordpress site where contributors can add their own tags.

I realize this thread may not be the place to discuss it.

3 Likes

Can you give an example how I would use it as a translator in Bilara?

2 Likes

You wouldn’t. It’s a separate task for a separate volunteer. So just keep going! At some point we might localize the tags, but that would just mean translating the list and their definitions.

Hasantha, who is interested to undertake this, will hopefully make a post here some time for discussing this. Meanwhile, I’ve been discussing it with him here (he’s in Sydney and I see him every week).

2 Likes

Indeed he hase…

4 Likes