Day 1 & 2: Final Presentation

On the first day of this week, I worked on finalizing my presentation and shared it with my mentor. After receiving his feedback, I made changes and focused on rehearsing the final presentation that I had to present to the entire team of HPCC systems (Richard Chapman, Vijay Raghavan, Michael Gardner, Lorraine Chapman, David de Hilster) along with other interns of summer 2022. On Aug. 2nd, I presented about the expectations that were set in the beginning and amount of work accomplished during the nine weeks period. I am very happy that I received a positive feedback and appreciation from entire team and my mentor. Since the presentation could not be recorded due to technical issues in Lorraine’s network I will soon record my presentation and share it to my manager, Lorraine and mentor, David.

Day 3, 4 & 5: Transforming KB to XML

Though I already presented my work of this summer I still have three more weeks of work remaining during which I have to accomplish remaining expectations that was set in deliverables before starting the project. Therefore, I started working on writing code in NLP++ to transform knowledge base developed earlier to XML format using which XML file of each Wiktionary word entry can be formed. For that, since it failed to spray data file in the ECL Watch, I am now writing and displaying out in word.xml file. With the help of my mentor, I learned to transform the knowledge base to XML. While doing so, I learned to use While con, strstartswith, and strval function to create a while loop, condition if string starts with string name such as “pronunciation”, “definition”, “synonym”, etc., and get the string value of concept built in knowledge base respectively. The final output of XML file was created and was tested on all input word files. It is writing XML file correctly for each word content except for the case when there are more than phonetic/phonemic in the pronunciation record. For example, the final output of word ” ” in the XML format is as follows:

<word>
	<wordid>1</wordid>
	<word>जवाफ</word>
</word>
<pos>
	<wordid>1</wordid>
	<posid>1</posid>
	<pos>नाम</pos>
</pos>
<definition>
	<posid>1</posid>
	<defid>1</defid>
</definition>
<explanation>
	<defid>1</defid>
	<expid>1</expid>
	<text>उत्तर; उत्तर पक्ष; उत्तरा; प्रतिवचन</text>
</explanation>
<variation>
	<defid>1</defid>
	<varid>1</varid>
	<text>उत्तर पक्ष</text>
</variation>
<variation>
	<defid>1</defid>
	<varid>2</varid>
	<text>उत्तरा</text>
</variation>
<variation>
	<defid>1</defid>
	<varid>3</varid>
	<text>प्रतिवचन</text>
</variation>
<example>
	<defid>1</defid>
	<exampid>1</exampid>
	<text>श्यामले आफ्नो प्रियसीलाई छोडेर जाने कारण सोध्दा प्रियसीले केहि जवाफ दीनन।</text>
</example>
<pronunciation>
	<wordid>1</wordid>
	<phoneticid>1</phoneticid>
	<pttext>javāpha</pttext>
</pronunciation>
<synonym>
	<wordid>1</wordid>
	<synid>1</synid>
	<text>उत्तर</text>
</synonym>
<synonym>
	<wordid>1</wordid>
	<synid>2</synid>
	<text>प्रतिक्रिया</text>
</synonym>
<term>
	<wordid>1</wordid>
	<termid>1</termid>
	<text>जवाफदेही</text>
</term>
<term>
	<wordid>1</wordid>
	<termid>2</termid>
	<text>जवाफमाग्नु</text>
</term>
<term>
	<wordid>1</wordid>
	<termid>3</termid>
	<text>जवाफदिनु</text>
</term>
<language>
	<wordid>1</wordid>
	<langid>1</langid>
	<text>अङ्ग्रेजी</text>
</language>
<translation>
	<langid>1</langid>
	<transid>1</transid>
	<text>answer</text>
</translation>
<translation>
	<langid>1</langid>
	<transid>2</transid>
	<text>reply</text>
</translation>
<translation>
	<langid>1</langid>
	<transid>3</transid>
	<text>response</text>
</translation>

As shown above, I could able to transform the knowledge base of each word input file to the output in XML format.

Day 1

After knowing all the bugs in the analyzer sequences, I went through the analyzer sequences and made changes to the respective sequences where needed. As a result, I could able to fix most of the bugs such as the pronunciation zone was showing its content correctly though there were only phonetic(s) or phonemic(s) or both, [[ ]] were removed from most sections in each input file. However, there was still [[ ]] in some sections such as variation and definition sections which I could not remove. Additionally, the definition count was still wrong and gave counts of all definitions of that word instead of the count of definitions that were under a particular part of speech.

Day 2

I continued working on fixing bugs. While debugging, I came to know that the hierarchy of content zones was wrong for definition. Therefore, I had to add an “explanation” zone under the definition that takes care of the text in the definition zone. To fix the number of count of definitions for the respective part of speech, I had to make changes in the code of kbDefText, kbDefVariation, and add kbExp for explanation section of definitions. As a result, I obtained a complete knowledge base of word’s content and one of the example of knowledge base structure is as follows:

Word: answer (जवाफ)

words
  जवाफ: 
    pos=[1]
    pos1: 
      pos=[नाम]
      definition=[1]
      definition1: 
        explanation=[1]
        variation=[3]
        example=[1]
        explanation1: 
          text=[उत्तर; उत्तर पक्ष; उत्तरा; प्रतिवचन]
        variation1: 
          text=[उत्तर पक्ष]
        variation2: 
          text=[उत्तरा]
        variation3: 
          text=[प्रतिवचन]
        example1: 
          text=[श्यामले आफ्नो प्रियसीलाई छोडेर जाने कारण सोध्दा प्रियसीले केहि जवाफ दीनन।]
    pronunciation: 
      phonetic=[javāpha,javaapha]
    synonym: 
      synonym=[2]
      synonym1: 
        text=[उत्तर]
      synonym2: 
        text=[प्रतिक्रिया]
    derivedTerms: 
      derived=[3]
      derived1: 
        derivedTerms=[जवाफदेही]
      derived2: 
        derivedTerms=[जवाफमाग्नु]
      derived3: 
        derivedTerms=[जवाफदिनु]
    translation
      अङ्ग्रेजी: 
        translation=[3]
        translation1: 
          text=[answer]
        translation2: 
          text=[reply]
        translation3: 
          text=[response]

Day 3,4 & 5

After completing building knowledge base in NLP++, the next step was to get HPCC systems running on my PC. For this, I installed Ubuntu on windows and followed the link: https://github.com/hpcc-systems/HPCC-Platform/wiki/Building-HPCC to install NodeJs, Ubuntu, and complete required installation. However, I could able to install vcpkg on Windows subsystems for Linux (WSL) but, when I was trying to run sudo cmake -j4 package the compilation was not completing to 100% and stopping in the middle giving error. This may be occuring because of very less memory space left in my PC. Therefore, I tried another way i.e. by installing Ubuntu and running it on virtual box. I received similar error as on WSL while trying to complete configuration for HPCC systems on Ubuntu on Virtual Box. I spent three days and even after receiving help from my mentor and another expert, Michael Gardner, I still haven’t been able to resolve the issue.

Day 1:

I completed parsing the Wiktionary data by building all the required analyzer sequences that match the pattern of word entry text in the Wiktionary word entry format.

Day 2 & 3:

I started to build a knowledge base (KB) by adding post areas in the existing analyzer sequences and even by building new analyzer sequences as needed. Some of the analyzer sequences I had to build to build a knowledge base are kbZones, kbPhonetics, kbPhonemics, kbDefText, kbVariations, kbExamples, kbDerivedTerms, and kbTranslations. These analyzer sequences help organize all the content of a word in the computer’s knowledge same as humans perceive these words’ content in their head. To illustrate, the following are codes written to build a knowledge base of kbZones that includes sub-zones of words such as pronunciation, synonyms, derived terms, and translations:

@NODES _ROOT

@POST
N("pronunciation") = makeconcept(G("word"),"pronunciation");

@RULES
_xNIL <-
	_pronunciations	### (1)
	@@

@POST
N("synonym") = makeconcept(G("word"),"synonym");

@RULES
_xNIL <-
	_synonyms	### (1)
	@@


@POST
N("derivedTerms") = makeconcept(G("word"),"derivedTerms");

@RULES
_xNIL <-
	_derivedTerms	### (1)
	@@

@POST
N("translation") = makeconcept(G("word"),"translation");

@RULES
_xNIL <-
	_translations	### (1)
	@@

In the above code, makeconcept creates concept in knowedge base for the particular zone of word’s content where text from each subzone will be added.

I also learned to use MakeCountCon which counts all the entries of a particular concept such as if there are more than one part of speech or multiple definitions in the def zone then it counts all and posts under the same zone i.e. pronunciation zone or def zone respectively. The code examples of MakeCountCon are as follows:

@NODES _posZone

@POST
N("con",1) = MakeCountCon(X("con"),"definition");

@RULES
_xNIL <-
	_defZone	### (1)
	@@

@PATH  _ROOT _posZone _defZone _definition _headerZone _LINE _item

@POST
L("con") = MakeCountCon(X("con",3),"variation");
addstrval(L("con"),"text",N("$text",2));
singler(2,2);

@RULES
_variation <-
	\;								### (1)
	_xWILD [plus fail=(\; । _xEND)]	### (2)
	@@

In the above code block, variation is parsed and added to the knowledge base using NLP++. we have defined variations as different forms of definitions or different ways to say the same meaning. In our knowledge base, variations are parts of the definition separated by “;” in the definition text and there can be one or more variations of definition so, I used MakeCountCon to count all the concepts of variations and post them in their respective zone.

In this way, I build all the knowledge base analyzer sequences and could add all the content of word in it.

Day 4 & 5

On the fourth day, I ran a newly developed analyzer (having analyzer sequences to parse the word content and zones along with knowledge base analyzer sequences) on all input .txt files of words to test if it works for all kinds of input data. While running the analyzer, I found some bugs such as those words having only phonetic/phonemic or more than one phonetics/phonemics was not showing in the output of knowledge base, definitions count was wrong for many words, [[ ]] was not removed from many word’s defText and variations which creates a link for the particular word that shows the very word in either red or green color (red shows that word is not added to Wiktionary yet whereas word in the green text means it is added to the Wiktionary and can be opened to view its content.

Planning for Remaining Weeks

I had a meeting with my mentor, David to plan and discuss about work to accomplish in the remaining weeks. During that meeting, we discussed about our targetted goal which is to build a dictionary in NLP++. For this, first, I will have to take the NeWiktionary analyzer that I have built and run it on HPCC systems, design a record structure in database format having word, definition, pronunciation, parts of speech, definitions, examples, derived terms, synonyms, and translations in English. To do this, I have to download Ubuntu on Windows, run HPCC systems on it, and finally build a dictionary for NLP++ from the Knowledge base of Wiktionary data.