monkeyunited

u/monkeyunited


r/dataengineering•

I'm using Python and Pyspark.

I inherited a pipeline that does the following:

import business_logic1, business_logic2

class main():
    _init_():
        self.table1 = "table1"
        self.table2 = "table2"
        if Boolean:
            self.table3 = "table_xxx"
        else:
            self.table3 = "table_yyy"
        ...
        # many many more tables
    
    def run():
        dict_business_logic1 = {
            "table1" = self.table1,
            "table5" = self.table5,
            "table7" = self.table7,
            ...
        }
        business_logic1 = business_logic1(dict_method1)
    
03

r/datascience•

I had just delivered a product where the pipeline was broken into a few parts, then given to developers to work on due to the short timeline.

I finally had a chance to look at my colleague's code and was taken aback by how un-pythonic it is. Some of the main offenders include:

  • not using "if __name__ == __main__"
  • everything is in classes where most classes have only one function (no "init" function in it, if that matters)
  • if it's a big class, the "main" function is really really really really long
  • No comments, no tests but I suppose those can be attributed to rushing to deliver

I just don't know if I should care. I don't need to maintain his code because my pipeline starts

646

r/datascience•

We have a NLP problem where some dataset contains some keywords while some don't. The key words are highly correlated to the outcomes so when the text has keywords, the model performs really well but not so much otherwise.

We are thinking about using weighted loss function, where dataset without keywords have a higher weight so model is penalized more for getting the predictions wrong.

Has anyone done this and is this a feasible route? I'm digging through research papers but figured it can't hurt to ask here.

It is obviously borrowing weighted loss function common used to handle imbalanced dataset, but in this case, it's not the class that's imbalanced but rather the features. We can also

15

r/datascience•

I have a full time job and took on a side gig which has grew and require more time than I'm willing to put in. The incremental responsibility is only 2-4 hours per week so not quite enough for me to expand my business but enough for me to take QOL hit because I constantly have the weight of unfinished tasks in the back of my mind.

The client has proposed to hire another contractor but I don't like it because I lose control over the project and I can make more by outsourcing the work.

However, I don't have experience with outsourcing and worry about having to manage someone else's work. I'm using client provided laptop so I can't use sites such as fiverr. It feels like the simplest way is f

18

r/statistics•

I'm running a Chi-Square test of independence to test if two categorical variables are related. Because of our sample size (3k to even 500k), the tests are always rejected which makes sense because even if an event is random, if it happens enough time the test would still consider it "too many times".

Edit: I want to emphasis that this is an expected behavior and I'm interested in how to properly handle this, rather than why this is happening. I knew this will happen.

I'm wondering what's the appropriate next step then? Is it to find a sample size, n, such that Chi-Square test is still efficient and just randomly draw samples up to n?

Or is there a similar test that's less affected by

110

r/datascience•

A previous employer of mine had reached out to ask for my help as an outside consultant. I'm interested but have no experience working as a consultant.

The work will be supporting operational report run, building and automating ETL processes, and likely analysis work. In other words, there will be weekly/monthly deliverables and mid-to-long term projects.

I don't know what's an appropriate rate. Is 1.5x my current hourly salary reasonable?

Also, I don't know if I need to create a company. Is that better for tax purposes/liability, or can I keep it simple and collect a monthly check (like back in the days when I do tutoring) and just claim it as ordinary income?

What else do I need to kno

212


r/orangecounty•

Can I get a recommendation on reliable and trustworthy handyman?

I need to replace a garbage disposal or unstuck it in Irvine. This is a for a rental place that would need help a few times a year.

34

r/learnpython•

Apologize if this is the wrong sub.

I have .rdl reports hosted on SSRS server. Currently the user has to click into them, select filters, click "View Report", and then select download as Excel/csv.

I've found no solution for Python to interact with SSRS directly. The only solutions I can find through googling are through "web-scrapping" methods instead of working with SSRS. For example: https://www.reddit.com/r/Python/comments/67xak4/enterprise_intranet_authentication_and_ssrs/

Does SSRS just don't work with Python? As in, it's currently not possible to do something like "from Python, conne

03

r/orangecounty•

Would really appreciate some perspectives on this.

Wife and I bought a condo and initially I was all about 30 year fixed, but the loan officer suggests us to look into 10/1 arm and, to my surprise, it kind of makes sense?

We're quoted 2.85% for 30 year fixed and 2.375% for 10/1 arm. Mortgage difference is about $150 - 200 a month, which we'll use as extra payment to mortgage. At 10 year mark, according to amortization table, 10/1 arm will be $30k less in balance.

It's our first home purchase for a three-story 4B 3.5B new construction condo in Brea. We like the area, school district, and size/room numbers are plenty if we have kids. In other words, we planned to stay long term when we buy

425

r/datascience•

I'm trying to understand if it's possible to use Confluence as self-servicing analytics tool.

My goal is to have a knowledge database where you can search for past analytics projects or routine processes. Within the page, there will be either github link or Excel spreadsheet attached to the page with instructions on filters and parameters so anyone with the right access can reproduce the process.

I chose Confluence because I also plan to set up a ticketing system using Jira. The Confluence page will have past Jira tickets attached to provide context.

First of all, do you think this is feasible? Will this scale well into the future or becomes a swamp to maintain? What are some of the poten

11

r/datascience•

TL;DR - what are some ways to track analytics performances? Need to make proposal to my boss, who had no experience in analytics.

Recently joined a small company with super old school management style - micro-managing, KPI with moving goalpost, no WFH, and all the problems associated with such management style such as office politics, lack of talent, high turnover, and apathy from lower level staff. My plan currently is accumulating enough time to not look like a hopper, then I'm out.

Earlier this week, my boss (head of OPs) sent a weekly goal of "delivering 3 analytics projects identified by xxx department". I feel this is something you say when you have no idea how analytics works.

I'm

010


r/orangecounty•

From Brea. Got a potential good offer and unfortunately WFH or moving is not an option in near future.

Has anyone done it? How bad is the commute?

Do you think having a adaptive cruise control or even Tesla FSD will make it tolerable?

1721