Chassidus1 offers the following explanation to resolve the second question mentioned aboveוּמוּבָא הַבֵּאוּר בָּזֶה
based on the Midrash2 that states:עַל פִּי מַה דְּאִיתָא בְּמִדְרָשׁ
“It would have been appropriate that the concluding festival of the holiday of Sukkos be removed 50 days from it like the concluding festival of Pesach (i.e., Shavuos).רְאוּיָה הָיְתָה הָעֲצֶרֶת שֶׁל חָג שֶׁתְּהֵא רְחוֹקָה חֲמִשִּׁים יוֹם כְּנֶגֶד הָעֲצֶרֶת שֶׁל פֶּסַח
There is, however, a difference between these two. With regard to Shemini Atzeres, the concluding festival of the holiday of Sukkos, the Jewish people are making a transition from summer to winter, andאֶלָּא עֲצֶרֶת שֶׁל חָג עַל יְדֵי שֶׁהֵן יוֹצְאִין מִן הַקַּיִץ לַחֹרֶף
as such they do not have a sufficient number of days to go and then return,”3 for the rainy season begins shortly after Sukkos.לֵית בְּיוֹמַיְיהוּ דְּיֵזְלוּן וְיֵיתוּן,
“To illustrate with an analogy: It is like a king who had many daughters,מָשָׁל לְמָה הַדָּבָר דּוֹמֶה לְמֶלֶךְ שֶׁהָיוּ לוֹ בָּנוֹת הַרְבֵּה,
some of whom were married and lived near their father, while others were married and lived in distant places.מֵהֶן נְשׂוּאוֹת לְמָקוֹם קָרוֹב וּמֵהֶן נְשׂוּאוֹת לְמָקוֹם רָחוֹק,
One day, they all came to inquire about the welfare of their father, the king.יוֹם אֶחָד בָּאוּ כֻּלָּן לִשְׁאוֹל שְׁלוֹם הַמֶּלֶךְ אֲבִיהֶם,
The king said: ‘Those of my daughters who are married and live nearby have ample opportunity to go and return;אָמַר הַמֶּלֶךְ אֵלּוּ שֶׁנְּשׂוּאוֹת בְּמָקוֹם קָרוֹב אִית בְּעוֹנָתָן לְמֵיזַל וּלְמֵיתֵי,
those who are married and live far away don’t have the opportunity to go and return soon.וְאִילֵין שֶׁנְּשׂוּאוֹת בְּמָקוֹם רָחוֹק לֵית בְּעוֹנָתָן לְמֵיזַל וּלְמֵיתֵי,
Now, while they are all still with me here, let us all make a festival of one day and rejoice together with them.’עַד דְּאִינוּן כֻּלָּן אֶצְלִי הָכָא נַעֲבֵיד כֻּלָּן חַד יוֹם-טוֹב וְנֶחֱדֵי עִמָּן,
Similarly, in the analogue: With regard to the festival following Pesach, when the Jews are making the transition from winter to summer, the Holy One, blessed be He, said, ‘They have sufficient opportunity to go and return.’כָּךְ עֲצֶרֶת שֶׁל פֶּסַח עַד דְּאִינוּן נָפְקִין מֵחֹרֶף לְקַיִץ אָמַר הקב"ה אִית בְּיוֹמָא לְמֵיזַל וּלְמֵיתֵי,
However, with regard to the festival following Sukkos, since they are making the transition from summer to winter,אֲבָל עֲצֶרֶת שֶׁל חָג עַל יְדֵי שֶׁהֵן יוֹצְאִין מֵהַקַּיִץ לַחֹרֶף כו'
the Holy One, blessed be He, says, ‘they do not have a sufficient number of days to go and return.’אָמַר הקב"ה לֵית בְּיוֹמָא לְמֵיזַל וּלְמֵיתֵי
As such, ‘While they are still here, let us all make a festival of one day and rejoice.’”אֶלָּא עַד דְּאִינוּן הָכָא נַעֲבֵיד כֻּלָּן חַד יוֹם-טוֹב וְנֶחֱדֵי.
This explains why Simchas Torah is celebrated on Shemini Atzeres.וְזֶהוּ מַה שֶּׁעוֹשִׂין שִׂמְחַת תּוֹרָה בִּשְׁמִינִי עֲצֶרֶת,
For the “one day” when all the “daughters” comeכִּי הַ"יּוֹם אֶחָד" שֶׁבּוֹ בָּאִים כָּל הַבָּנוֹת
[i.e., the daughters of Zion and Jerusalem, the souls of the Jewish people4][בְּנוֹת צִיּוֹן וִירוּשָׁלַיִם, הַיְנוּ נִשְׁמוֹת יִשְׂרָאֵל]
to inquire as to the welfare of their Father, the King,לִשְׁאוֹל שְׁלוֹם הַמֶּלֶךְ אֲבִיהֶם
refers to Yom Kippur.5קָאֵי עַל יוֹם הַכִּפֻּרִים,
Afterwards, the King says, “Let us all make a festival of one day and rejoice together”;4וּלְאַחֲרֵי זֶה אוֹמֵר נַעֲבֵיד כֻּלָּן חַד יוֹם-טוֹב וְנֶחֱדֵי,
this refers to Simchas Torah,הוּא שִׂמְחַת תּוֹרָה
(which was ordained by our Sages to be celebrated on Shemini Atzeres),(שֶׁנִּקְבַּע בִּשְׁמִינִי עֲצֶרֶת),
and is referred to as “one festival,” similar to the “one festival” of Yom Kippur.שֶׁהוּא "חַד יוֹם-טוֹב" בְּדֻגְמַת הַ"יּוֹם אֶחָד" דְּיוֹם הַכִּפֻּרִים.
As will presently be explained, Yom Kippur and Simchas Torah share a commonality in that they both express the essential dimension of the Jewish soul. Therefore, “all the daughters,” the entire Jewish people, are aroused on those days.

Summary

The Midrash explains the reason that Shemini Atzeres is celebrated directly after Sukkos (in contrast to Shavuos, which is celebrated 50 days after Pesach) with an analogy of a king, whose daughters all come from far and near to visit their father on one day, which refers to Yom Kippur. In appreciation of their visit, the king decides to establish one day of celebration – Simchas Torah – to rejoice together with them.